On this page 16
An always-on AI agent keeps working toward a defined goal between user sessions. To build one, persist its tasks and decisions, wake it when an event or deadline matters, give it only the context it needs, and require authorization before consequential actions. The model does not need to run continuously; the system needs to remember reliably.
That distinction changes the architecture. You are not building a chatbot with a longer timeout. You are building an event-driven application that uses a language model for selected decisions, while ordinary software controls scheduling, permissions, retries and execution.
This guide develops a provider-neutral design around a support-inbox agent. It includes PostgreSQL worker-claim examples, a runnable Node.js approval-binding exercise, recovery rules, a cost explorer and a failure-testing matrix. The examples illustrate specific boundaries; they are not a complete deployed agent or a claim of production qualification.
What makes an AI agent always-on?
The useful definition is durable identity, durable work and controlled wake-ups. Cloudflare's long-running-agent documentation explicitly distinguishes a persistent agent identity from a continuously running process: durable state can survive restarts, while ordinary in-memory variables and timers cannot.1
There are several different problems hiding behind the word “background”:
| Mechanism | What it provides | What you still need |
|---|---|---|
| A chat request | An interactive response | Work that outlives the request |
| A scheduled job | A trigger at a configured time | Task history, recovery and authorization |
| A background model request | Asynchronous model execution | Application-level workflow state and tool safety |
| A durable agent workflow | Persistent progress across interruptions | Good decisions, bounded permissions and verified outcomes |
For example, OpenAI's Responses API supports background response generation and completion notifications through webhooks. Those are useful building blocks, but enabling background: true is not the same as implementing your application's approval records, event subscriptions or external-action recovery.23
A durable workflow engine addresses another layer: preserving execution progress through failures and waits. Temporal's AI documentation describes this layer for agents and human approvals.4
Keep the distinction explicit: the model decides what might help; the application decides what is allowed and records what actually happened.
Start with one bounded job
Consider this user request:
Watch the connected support inbox. When a new customer message needs attention, find relevant account information, draft a reply and ask me before sending it. Remind me if I have not reviewed it within two hours.
This is deliberately narrower than “manage my support operations.” It gives the agent a useful responsibility without handing it an undefined mandate.
Before choosing a framework, translate the request into an application contract:
| Decision | Contract for this example |
|---|---|
| Activation | A verified mailbox change or a durable review deadline |
| Allowed reading | The connected inbox and authorized account documents |
| Model responsibility | Interpret the message, retrieve evidence and propose a draft |
| Sending permission | Approval for one exact reply through one connected account |
| Success evidence | Provider acceptance recorded against the approved action |
| Stop conditions | Cancellation, revoked access, expired approval or exhausted budget |
“Provider accepted the message” is not the same as “the customer read it.” Make the completion label match the evidence your connector can actually provide.
The reminder itself does not need an LLM. Once a review deadline exists, ordinary code can decide whether it has passed. Likewise, tenant selection, credential selection and authorization must not be model guesses.
Use AI where interpretation adds value. Keep predictable control decisions deterministic. This is the design recommendation used throughout the article, not a requirement to adopt a particular agent SDK.
Design the architecture around durable work
The following design separates a durable control layer from replaceable model and tool execution. The boxes represent responsibilities; they do not require nine independently deployed services.
Diagram source
flowchart TD
accTitle: An event-driven always-on agent
accDescr: Verified events and durable deadlines enter a persistent inbox. A worker loads authorized context, asks the model for a proposal and routes it through policy and approval checks before a connector can act. Recorded results feed the user interface.
Events[Webhooks and durable deadlines] --> Admission[Authenticate and deduplicate]
Admission --> Store[(Inbox and run state)]
Store --> Worker[Worker claims a bounded step]
Worker --> Context[Load scoped evidence]
Context --> Model[Model proposes the next action]
Model --> Gate{Policy and approval gate}
Gate -->|Needs review| Approval[Persist approval request]
Approval --> Store
Gate -->|Authorized| Tool[Execute through connector]
Tool --> Ledger[(Action result and audit record)]
Ledger --> Store
Ledger --> UI[Progress and notifications]The crucial boundary is between proposal and execution. A tool call emitted by a model is input to your application, not permission to bypass its controls.
For an initial deployment, one backend, one worker process, PostgreSQL and an existing durable job mechanism can cover these responsibilities. Keep the components together until measured load, isolation or operational requirements justify separation.
Persist enough information to resume the work without reconstructing truth from a chat transcript: the tenant and principal, goal version, source event, current checkpoint, pending action, approval reference and last verified result. Keep model, prompt and tool-schema versions alongside the run so a later incident can be reproduced.
An agent that wakes tomorrow should answer three questions from durable records: What am I doing? What has already happened? What am I permitted to do next?
Trigger work without repeatedly asking the model whether anything changed
An event should first pass through inexpensive deterministic checks. Authenticate the sender, validate the envelope, resolve the connected account to its tenant and reject irrelevant or duplicate deliveries before invoking a model.
Make event admission atomic
Use a stable uniqueness boundary such as (tenant_id, connection_id, provider_event_id). The tenant and connection come from a trusted server-side mapping, not an arbitrary field in a webhook payload.
In one database transaction, insert the accepted event and the durable work it creates. If delivery to another system is required, record an outbox message in that same transaction. Acknowledge the webhook only after acceptance is durable. If a duplicate arrives, acknowledge it without creating another business operation.
The transactional outbox pattern prevents the database-update/message-publication gap. It does not eliminate duplicate delivery; consumers still need deduplication.5
Distinguish three identities:
| Identifier | Purpose |
|---|---|
| Event ID | Deduplicate a provider delivery |
| Business-operation ID | Prevent two events from creating the same logical reply or reminder |
| Attempt ID | Diagnose a particular execution attempt |
Do not use the attempt ID as the external action's idempotency key. A retry is another attempt at the same action, not a new action.
Treat mailbox notifications as change signals
Gmail is a useful concrete example. Its push integration uses Cloud Pub/Sub; a notification identifies mailbox history to inspect rather than delivering a complete, ready-to-use message. Watches require renewal, and notifications can be delayed or dropped. Google's documentation explicitly recommends a recovery synchronization path.6
Therefore, persist the synchronization cursor, renew subscriptions before expiry and fetch changes through the provider's supported history mechanism. If a cursor is no longer valid, follow the provider's resynchronization procedure instead of silently skipping the gap.7
Prevent feedback loops as well: the agent's own sent reply may produce another mailbox event. Recognize its recorded action or message ID and apply deterministic rules before creating a new run.
Persist deadlines, not just timers
For “remind me in two hours,” store the deadline durably. For “every weekday at 9 a.m.,” also store the intended time zone, local-time rule and policy for missed occurrences. Decide whether downtime should skip, coalesce or replay overdue jobs.
Use a uniqueness key for each scheduled occurrence. An in-memory timer can help a live process wake up, but the stored schedule must be sufficient to recover after a restart.
Give every run an explicit state
A run should not oscillate vaguely between “thinking” and “done.” Make the reason it is active, waiting or stopped explicit.
Diagram source
stateDiagram-v2
accTitle: Durable run lifecycle
accDescr: A run can execute, wait for approval, wait for a safe retry or reconcile an uncertain result. Success requires verified completion. Cancellation and failures stop further work without erasing past effects.
[*] --> Queued
Queued --> Running: Claim available work
Running --> WaitingApproval: Persist exact proposal
WaitingApproval --> Queued: Valid approval arrives
WaitingApproval --> Stopped: Rejected or expired
Running --> RetryWait: Retry is safe
RetryWait --> Queued: Durable deadline arrives
Running --> Reconciling: External outcome unknown
Reconciling --> Queued: Safe continuation established
Reconciling --> Stopped: Needs manual resolution
Running --> Succeeded: Completion evidence recorded
Queued --> Stopped: Cancelled
Running --> Stopped: Cancelled or terminal failure
Succeeded --> [*]
Stopped --> [*]This is a conceptual lifecycle, not an exhaustive transition specification. For example, cancellation must also be handled while waiting or reconciling. Store a stop reason so rejection, cancellation, budget exhaustion and permanent failure remain distinguishable.
The run and its external actions need separate records. A stopped run might contain an already-sent email. Reporting only cancelled would hide that important fact.
A compact persistence model for this example is:
| Record | Durable information |
|---|---|
| Agent definition | Goal, owner, allowed connections, policy and configuration version |
| Run | Trigger, checkpoint, state, deadline, cancellation request and execution ownership |
| Action | Exact intended operation, stable identity, dispatch state and provider receipt |
| Approval | Bound action, decision, approver, expiry and consumption state |
| Evidence | Source references, versions, timestamps and access scope |
These are conceptual records, not a mandate to build five new storage services.
For a recurring agent, use bounded runs under a persistent agent identity. Retain the history you need; do not grow one unbounded conversation forever.
Claim work safely with PostgreSQL
Multiple workers must not casually execute the same step. PostgreSQL supports FOR UPDATE SKIP LOCKED for queue-like consumption: a worker can skip rows already locked by another consumer. PostgreSQL also warns that this produces an inconsistent view, so it is not a general-purpose query pattern.8
For a custom PostgreSQL-backed worker, a lease represents temporary execution ownership. A monotonically increasing claim epoch identifies which claimant is current. Both are application-level design choices in the following example.
Minimal schema for the worker-claim example
This table supports the SQL examples below. It deliberately omits the event, action and approval tables, authentication, migrations, retention and recovery scheduler; it is not a complete agent database.
CREATE TABLE agent_runs (
id uuid PRIMARY KEY,
tenant_id uuid NOT NULL,
state text NOT NULL DEFAULT 'queued'
CHECK (state IN (
'queued', 'running', 'waiting_approval', 'retry_wait',
'reconciling', 'succeeded', 'stopped'
)),
available_at timestamptz NOT NULL DEFAULT now(),
claim_epoch bigint NOT NULL DEFAULT 0 CHECK (claim_epoch >= 0),
lease_until timestamptz,
cancel_requested_at timestamptz,
checkpoint jsonb NOT NULL DEFAULT '{}'::jsonb
);
CREATE INDEX agent_runs_ready_idx
ON agent_runs (tenant_id, available_at, id)
WHERE state IN ('queued', 'retry_wait')
AND cancel_requested_at IS NULL;
The parameter $1 below is a tenant UUID selected by an authorized scheduler. Bind it through the database driver; never interpolate it into SQL. Execute the claim in a short transaction and commit before any model or network call.
WITH next_run AS (
SELECT id
FROM agent_runs
WHERE tenant_id = $1
AND state IN ('queued', 'retry_wait')
AND available_at <= clock_timestamp()
AND cancel_requested_at IS NULL
ORDER BY available_at, id
FOR UPDATE SKIP LOCKED
LIMIT 1
)
UPDATE agent_runs AS r
SET state = 'running',
claim_epoch = r.claim_epoch + 1,
lease_until = clock_timestamp() + interval '60 seconds'
FROM next_run
WHERE r.id = next_run.id
RETURNING r.id, r.tenant_id, r.claim_epoch, r.lease_until;
The 60-second lease is a teaching value, not a production recommendation. Choose lease and heartbeat behavior around actual step durations, failure detection and provider timeouts.
Checkpoint updates must be conditional on the same ownership. Here $2 is the run UUID, $3 its claimed epoch and $4 the serialized checkpoint:
UPDATE agent_runs
SET checkpoint = $4::jsonb,
state = 'waiting_approval',
lease_until = NULL
WHERE tenant_id = $1
AND id = $2
AND claim_epoch = $3
AND state = 'running'
AND lease_until > clock_timestamp()
AND cancel_requested_at IS NULL
RETURNING id;
If this returns no row, the worker must not treat the checkpoint as accepted. It may have lost ownership, expired or been cancelled. In the real application, persist the approval request and checkpoint in the same transaction.
A recovery mechanism must find expired claims and inspect their action records before making them runnable again. Lease expiry is not evidence that an external request failed. Similarly, an epoch check protects your database from stale updates; it cannot prevent an already-dispatched remote request from completing.
For wake-ups, a durable queue or workflow engine can carry the signal. PostgreSQL LISTEN/NOTIFY can provide a hint, but it is not a durable work ledger: only listening sessions receive notifications. After reconnecting, commit LISTEN before inspecting persisted work in a new transaction; recover missed deadlines from stored state. Use a session-capable dedicated connection for the listener rather than assuming a transaction-pooled connection preserves its registration.910
Recover external actions without blindly retrying
The hardest failure is not “the API returned an error.” It is “the API may have succeeded, but we did not record the result.”
Temporal's documentation describes this exact class of failure: an activity can complete its external work, then crash before recording completion. Retried execution can duplicate side effects unless the receiving service enforces idempotency.11
Diagram source
sequenceDiagram
accTitle: The uncertain-outcome gap
accDescr: An external service can complete an action before the worker loses its response. Recovery must inspect durable intent and provider evidence rather than assume the action failed.
participant Worker
participant DB as Action ledger
participant API as External service
Worker->>DB: Record immutable action and dispatch intent
Worker->>API: Submit action with stable identity
API->>API: Apply the action
Note over Worker,API: Response is lost or worker crashes
Worker->>DB: Restart and inspect unfinished action
Worker->>API: Reconcile using supported lookup or idempotency
API-->>Worker: Confirmed result or still unknown
Worker->>DB: Record evidence or request manual resolutionBefore dispatch, record the immutable action, its logical ID, exact request digest and any supported provider idempotency key. After dispatch, record the provider operation ID and the strongest completion evidence available. AWS's idempotent-API guidance explains why a caller-provided operation identity and consistent request intent matter for safe retries.12
Choose recovery by evidence:
| Observed situation | Safe next step |
|---|---|
| No dispatch could have occurred | Recheck policy and execute the pending action |
| Provider confirms success | Record the receipt; do not send again |
| Provider guarantees deduplication for the same key and payload | Retry that same action within the provider's supported window |
| Provider confirms no operation occurred, with a reliable lookup contract | Recheck authorization and consider another attempt |
| Timeout with no reliable deduplication or lookup | Keep the outcome unknown and escalate |
A missing search result is not necessarily proof of failure: indexes and asynchronous operations may lag. Follow the actual provider contract. For a concrete contrast, Stripe API v1 documents matching request parameters and a finite retention window for idempotency keys; a key reused after pruning can represent a new request. Do not generalize that contract to your email connector.13
For email, do not invent an exactly-once guarantee. A message header or your local unique constraint does not, by itself, make a mail provider deduplicate sends. When your connector cannot resolve an uncertain send safely, show the uncertainty to the operator instead of sending a second copy automatically.
For operations that are safe to retry, use a bounded retry policy with capped backoff and jitter, and respect provider retry guidance. Avoid independent retry loops at every layer: attempts can multiply. Permission failures, changed arguments and uncertain non-idempotent writes need different handling from transient read failures.14
Cancellation has a similar boundary. Stop authorizing new dispatches after cancellation is accepted. Attempt cancellation of in-flight work where supported, then reconcile its outcome. Never promise to unsend an email merely because a run was cancelled.
Separate workflow state from agent memory
“Add memory” is not a sufficient persistence design. The agent needs several kinds of information, with different authority and freshness rules.
| Information | Example | Appropriate treatment |
|---|---|---|
| Workflow state | This reply is awaiting approval | Transactional application record |
| Task evidence | A customer's current subscription details | Scoped source reference with freshness checks |
| User preference | Prefer concise replies | Editable preference with provenance |
| Working context | The message and evidence for this decision | Small, temporary model input |
| Prior summary | What an earlier session attempted | Navigational aid, not proof of completion |
Anthropic's context-engineering guidance emphasizes selecting relevant information rather than continually expanding the prompt. Its long-running-agent research also shows why progress artifacts matter across sessions: context compaction alone does not establish that work is complete.1516
For the inbox agent, construct each model request from the current goal, relevant message, authorized evidence, applicable rules and remaining budget. Do not attach the entire mailbox or every previous run.
A practical memory record should identify its tenant and subject, source reference, observation time, validity or expiry, and whether it was explicitly provided or inferred. An inference such as “the customer probably prefers refunds” must not silently become an account policy.
On retrieval, check access rights before supplying information to the model. Revalidate time-sensitive facts before an action. If a customer changes their address after a draft was created, a stored summary is not an excuse to use the old address.
Compaction, clearing replaceable tool results and cross-session memory are separate mechanisms in Anthropic's context-management examples. Treat a compacted summary as a useful working representation, while keeping authoritative actions, approvals and source evidence outside it.17
Memory deletion should include derived records and retrieval indexes, not only the visible conversation. Keep necessary audit retention separate, access-controlled and documented rather than promising that deleting a chat erases every operational record.
Bind human approval to one exact action
A generic “Allow this agent” button is too broad for the support example. The reviewer needs to see the sending account, recipient, reply content, relevant context and consequences.
OWASP's transaction-authorization guidance recommends binding authorization to significant transaction data, invalidating it when that data changes and enforcing a final authorization gate before execution. Those principles apply directly to agent-generated actions.18
For this design, approval should bind to:
Tenant + run + action + connected account + tool version + exact arguments + relevant source version + policy version + expiry.
The execution gate must reload trusted records and check the approver's current authority, current connection scope, cancellation state and action identity. A statement inside a message or model response saying “the user approved” is not an approval record.
If the draft or destination changes, request approval again. If new information makes the draft stale, stop and review it. A source-version check detects known changes, but without provider-supported conditional execution it cannot eliminate every race between the final read and an external write. Document that residual risk.
A small, runnable Node.js approval-binding exercise
The following utility demonstrates one narrow invariant: changing any execution-relevant field changes the action digest. It uses a fixed field order and rejects unknown fields so an unbound bcc field cannot quietly alter the operation.
mail.send_reply.v1 and threadVersion are application-defined fields, not Gmail API parameters. Define a source-version mechanism appropriate to your connector. This intentionally supports only a nonempty, plain-text reply with the fields shown. Expand the schema and its version deliberately before adding attachments, HTML, CC/BCC or other execution parameters.
Copy action-digest.mjs
import { createHash } from 'node:crypto';
// This deliberately small schema supports one plain-text reply only.
const fields = Object.freeze([
'tenantId', 'runId', 'actionId', 'connectionId', 'tool',
'recipient', 'threadId', 'threadVersion', 'subject', 'body',
'policyVersion'
]);
export function actionDigest(action) {
if (!action || typeof action !== 'object' || Array.isArray(action)) {
throw new TypeError('Expected an action object');
}
const keys = Object.keys(action);
if (keys.length !== fields.length ||
keys.some(key => !fields.includes(key)) ||
fields.some(key => !Object.hasOwn(action, key) ||
typeof action[key] !== 'string' || !action[key].trim())) {
throw new TypeError('Unexpected, missing, or empty action field');
}
if (action.tool !== 'mail.send_reply.v1') {
throw new TypeError('Unsupported tool');
}
// Fixed field order; preserve the exact approved text, including whitespace.
const bytes = JSON.stringify(['approval-v1', ...fields.map(key => action[key])]);
return createHash('sha256').update(bytes, 'utf8').digest('hex');
}
Copy action-digest.test.mjs and run the checks
Save this file beside action-digest.mjs, then run node --test action-digest.test.mjs with a supported Node.js installation. No third-party package is required for this exercise.
import test from 'node:test';
import assert from 'node:assert/strict';
import { actionDigest } from './action-digest.mjs';
const action = Object.freeze({
tenantId: 'tenant-a', runId: 'run-41', actionId: 'action-9',
connectionId: 'support-inbox-a', tool: 'mail.send_reply.v1',
recipient: 'customer@example.com', threadId: 'thread-8',
threadVersion: '17', subject: 'Your support request',
body: 'We are investigating your request.', policyVersion: '3'
});
for (const field of [
'tenantId', 'runId', 'actionId', 'connectionId', 'recipient',
'threadId', 'threadVersion', 'subject', 'body', 'policyVersion'
]) {
test(`changing ${field} invalidates the approved digest`, () => {
const changed = { ...action, [field]: `${action[field]}-changed` };
assert.notEqual(actionDigest(changed), actionDigest(action));
});
}
test('object property order does not change the digest', () => {
const reordered = Object.fromEntries(Object.entries(action).reverse());
assert.equal(actionDigest(reordered), actionDigest(action));
});
test('unsupported fields cannot bypass approval binding', () => {
assert.throws(() => actionDigest({ ...action, bcc: 'other@example.com' }));
});
test('missing fields fail closed', () => {
const { connectionId, ...incomplete } = action;
assert.throws(() => actionDigest(incomplete));
});
test('a different tool fails closed', () => {
assert.throws(() => actionDigest({ ...action, tool: 'mail.delete' }));
});
test('whitespace changes remain visible to the digest', () => {
assert.notEqual(actionDigest({ ...action, body: `${action.body}\n` }),
actionDigest(action));
});
The digest is not a signature, credential, address validator or complete approval system. Store approval records on the server; never trust a browser-supplied digest as proof of consent. Authorize the reviewer independently, enforce expiry and atomically reserve the approved action so two concurrent clicks cannot create two sends.
After reservation, retries belong to that same immutable action. They must not create a fresh operation simply because the approval record has already been consumed. The connector must dispatch the exact fields that were approved, without post-approval transformations that change their meaning.
Put tool permissions outside the prompt
The inbox agent reads untrusted material by design. A customer email, retrieved page or attachment can contain instructions aimed at the model. OpenAI describes prompt injection as third-party content attempting to redirect an assistant away from the user's intended task, and recommends layered protections rather than assuming a single defense is sufficient.19
A stronger instruction such as “never do anything unsafe” is useful guidance, but it is not an authorization system.
Diagram source
flowchart TD
accTitle: Separate untrusted content from execution authority
accDescr: Email and retrieved text can inform a model proposal but cannot grant permissions. Trusted identity, policy and approval records independently control the execution gate. Credentials remain behind the connector boundary.
Content[Untrusted email and retrieved text] --> Proposal[Model proposal]
Proposal --> Gate{Server execution gate}
Identity[Trusted tenant and user identity] --> Gate
Policy[Current policy and tool allowlist] --> Gate
Approval[Server approval record] --> Gate
Gate -->|Allow exact action| Connector[Scoped connector]
Secrets[Credential vault] --> Connector
Connector --> API[External API]
Gate -->|Deny or pause| Status[Explain status to the user]For this reference design, enforce a small tool allowlist, validate arguments and bind resource access to the authenticated tenant. Keep OAuth tokens in the connector layer, out of model context and ordinary traces. Recheck authorization on execution, not only when the agent was created; OWASP's authorization guidance recommends validating permissions on every request.20
MCP does not remove these responsibilities. Its authorization specification requires token audience validation and separates a token presented to an MCP server from the credentials that server uses for upstream APIs. Do not pass an incoming token through indiscriminately.21
Tenant isolation applies to memory, retrieved documents, approval endpoints, action receipts and progress streams—not just the main task table. PostgreSQL row-level security can add protection, but table owners and privileged roles have important bypass behavior; test with the actual application role.22
For arbitrary code or browser execution, use an isolated environment with resource limits, restricted filesystem access and controlled network destinations. Protect server-side fetch tools against internal-network access and unexpected redirects. Do not introduce a browser when a narrow API operation solves the job more safely.
These controls reduce exposure; they do not establish that prompt injection is solved. Test malicious content through the complete workflow, including attempts to move instructions into memory or exploit a more privileged tool.23
Budget wake-ups, model calls and external work separately
An always-on agent can be quiet most of the day. Budgeting it as a model that must think every few minutes is often the wrong starting assumption.
Here is a deliberately simple arithmetic scenario, not a measured benchmark. One account receives 20 relevant events per day. Compare one model decision per event with an alternative that calls a model every five minutes to check for work. Use a 30-day month.
The periodic design makes 12 × 24 × 30 = 8,640 checking calls. The event-triggered design makes 20 × 30 = 600 decision calls. Subsequent task calls, retries and reconciliation are excluded from both counts. An ordinary connector sync does not inherently require a model call.
1 of 1 series shown
View data
| Illustrative activation policy | Calculated calls |
|---|---|
| Five-minute model checks | 8640 |
| Twenty relevant events per day | 600 |
The lesson is not a universal savings percentage. Remove unnecessary inference first, then measure the actual workload. An event-heavy system with expensive downstream reasoning may still cost more than this simple example.
Explore a transparent monthly cost model
The calculator below uses synthetic rates of $2 per million input tokens, $8 per million output tokens and a $20 fixed monthly infrastructure allowance. They are teaching assumptions, not prices from any provider or a capacity recommendation.
The starting volumes are 2.4 million input tokens and 0.24 million output tokens: 600 calls at an assumed 4,000 input and 400 output tokens each. That gives $6.72 for inference and $26.72 including the fixed allowance.
Here and are token quantities in millions, and are their fixed illustrative rates, and is the infrastructure allowance.
The chart is a fixed comparison; changing this calculator does not update it. Even with both quantities at zero, the example retains its $20 allowance. No model inference while waiting does not mean all infrastructure is free.
In a real system, meter the provider's billable usage categories, tool charges and compute separately. Enforce per-run deadlines, model-call limits, tool-call limits and concurrency limits in code. Reserve budget before admitting concurrent work, settle against actual usage and stop cleanly when the allowance is exhausted.
A useful operational measure is total attributable cost divided by verified successful tasks. Define the task population and time window, include failed attempts, and report the denominator. Cheap runs that accomplish nothing are not an optimization.
Make progress observable and completion verifiable
The user should be able to distinguish “queued,” “waiting for your approval,” “retry scheduled,” “access revoked” and “outcome unknown.” A spinner cannot communicate those differences.
Give each run a durable status and an ordered event history for user-facing progress. Deliver updates with server-sent events or WebSockets, and support a durable cursor or fresh snapshot after reconnection. Losing a browser connection must not cancel work unless that is an explicit product rule.
Correlate execution with tenant, run, action and attempt IDs. Instrument model calls and tools as distinct operations rather than treating the whole agent as an opaque request. OpenTelemetry's GenAI conventions provide a shared vocabulary for agent and tool spans; pin the convention version you adopt rather than assuming every attribute remains unchanged.24
Prefer short execution spans linked by run identity over keeping one operational trace open through days of human waiting. Keep sensitive payloads behind access controls and retention rules. Traces support diagnosis; the authoritative action and approval records must not disappear when traces are sampled out.
Test outcomes, not the agent's confidence
Anthropic's evaluation guidance distinguishes the transcript from the outcome: an agent saying an operation succeeded is different from the corresponding state actually existing. It also recommends using appropriate combinations of deterministic, model-based and human grading.25
For this inbox example, code can verify the approved recipient, action count, source version and provider receipt. Human review or a calibrated model judge can assess whether the draft addresses the customer accurately. Do not let a fluent draft compensate for a permission violation.
Use a failure matrix before expanding autonomy:
| Test | What to inject | Required evidence |
|---|---|---|
| Duplicate event | Deliver the same event repeatedly | One logical operation; duplicate deliveries recorded |
| Lost response | Accept the send, then drop the response | No blind resend; reconciliation or explicit uncertainty |
| Worker replacement | Resume an expired worker after a new claim | Stale checkpoint rejected; remote effects reconciled |
| Changed approval | Modify recipient, body or account after review | Old approval cannot authorize the changed action |
| Revoked access | Disconnect the account while approval is pending | No new dispatch; user sees the actual blocked reason |
| Malicious evidence | Put instructions in a customer message or document | No extra authority, secret exposure or cross-tenant retrieval |
| Missing notification | Drop a change event or expire the subscription | Recovery synchronization finds the missed work |
| Cancellation race | Cancel immediately before and after dispatch | No new authorized dispatch; in-flight result represented honestly |
| Concurrent work | Flood one tenant and one shared conversation | Bounded resources and no conflicting duplicate replies |
| Budget exhaustion | Reach the limit during a multi-step task | Durable pause or stop, without an unbounded retry loop |
Run these against real adapters in a controlled test environment, not only mocked functions. The digest exercise above verifies one local invariant; it does not prove any of these end-to-end behaviors.
Record which scenarios ran, what failed and what remains untested. Preserve failing cases as regression fixtures when the model, prompt, connector or policy changes.
Choose a runtime without building unnecessary infrastructure
There is no single required deployment stack. The appropriate choice depends on the work you must make durable and what your team already operates.
| Starting point | Reasonable fit | Responsibility you still own |
|---|---|---|
| Existing backend and durable job system | A few bounded workflows with familiar operations | Business state, permissions and safe side effects |
| PostgreSQL-backed custom worker | Small, well-understood execution needs | Leases, timers, recovery, fairness and operational tooling |
| Temporal or a comparable workflow engine | Long waits and complex resumable work | Idempotent activities and business authorization |
| Cloudflare Agents with Workflows | An edge-oriented deployment using that platform | Application policy, connector behavior and vendor-specific limits |
Temporal documents durable AI workflows; Cloudflare documents an agent/workflow split and a durable-agent tutorial. These are implementation options, not prerequisites for the provider-neutral design in this article.42627
Do not build a custom distributed scheduler merely to avoid a workflow dependency. Equally, do not introduce several orchestration systems for a single bounded job. Choose one owner for durable execution and reuse the infrastructure you can operate well.
Before adding more agents, verify backup restoration, deployment compatibility for old checkpoints, tenant fairness and alerting for stuck work. Version long-lived payloads so a deployment does not make yesterday's pending approval unreadable.
For shared conversations, coordinate work at the thread or business-object boundary where conflicting actions matter. A global lock limits throughput unnecessarily; no coordination can produce two individually valid but contradictory replies.
A sensible first release is one workflow, a narrow toolset and approval-required writes. Increase autonomy only after failure testing and outcome evidence justify it.
Frequently asked questions
Does an always-on AI agent need a model running 24/7?
No. In this architecture, durable records and event delivery preserve continuity. The model runs for specific interpretation or planning steps. A worker or hosted platform may still have infrastructure costs while the agent is idle; “no inference” and “no cost” are different claims.1
Can I build this with Node.js and PostgreSQL without an agent framework?
Yes: the control flow, persistence and authorization in this reference design are ordinary application concerns. The tradeoff is ownership. Without a durable runtime, your application must implement and operate recovery, timers, execution claims and observability. A framework is optional; those reliability requirements are not. The SQL and Node.js examples cover individual boundaries, not the whole application.
Is long-term memory the same as durable execution?
No. A remembered preference or conversation summary does not tell a worker whether an email already sent, whether an approval expired or which execution attempt owns a task. Keep those facts in authoritative workflow and action records. Use memory to improve decisions, not to replace transaction state.
Can a human approval make an action completely safe?
No. The approval must be informed, bound to the exact action, unexpired and checked at execution time. Current permissions, changing external state and uncertain remote outcomes still matter. Approval is one control in a larger system, not a guarantee against every failure.18
The first milestone worth shipping
Build one complete path: receive a verified event, create a durable run, retrieve scoped evidence, propose a reply, persist approval, execute through a controlled connector and record the provider's result.
Then deliberately interrupt it. Restart the worker during approval. Redeliver the event. Revoke the connection. Lose the response after the external action. Resume the browser on another device. The system should preserve what it knows, avoid inventing what it does not know and show the user a truthful state.
An always-on AI agent is not a model that never stops thinking. It is a system that can stop, remember, resume and act within the authority it actually has.
Sources and scope
Primary documentation was consulted on October 29, 2026. The architecture, schema, digest exercise and workload model are original reference examples synthesized for this guide. Vendor documentation supports the cited capabilities and constraints; it does not certify this reference design. Numerical scenarios are explicitly illustrative, and no live model, mail provider or production deployment is claimed to have been benchmarked here.
Footnotes
-
Cloudflare, Long-running agents. Durable identities, activation and persistence boundaries. ↩ ↩2
-
OpenAI, Background mode. Asynchronous response generation; consult current data-control documentation for retention requirements specific to your account and usage. ↩
-
OpenAI, Webhooks. Completion notifications, signature verification and duplicate-delivery handling. ↩
-
Temporal, Durable AI. Agent workflows, approvals and durable execution options. ↩ ↩2
-
AWS, Transactional outbox pattern. The database/message dual-write problem and duplicate-message considerations. ↩
-
Google, Configure push notifications with the Gmail API. Mailbox watches, history cursors, renewal and delivery limitations. ↩
-
Google, Synchronize clients with Gmail. Partial synchronization and recovery when a history cursor is unavailable. ↩
-
PostgreSQL, SELECT: the locking clause. Queue-like use of
SKIP LOCKEDand its limitations. ↩ -
PostgreSQL, NOTIFY. Notification delivery and transaction semantics. ↩
-
PostgreSQL, LISTEN. Session-bound listeners and the initial-listening race. ↩
-
Temporal, Activity Definition. Idempotency, retries and completion-recording failures. ↩
-
AWS Builders' Library, Making retries safe with idempotent APIs. Stable caller intent, request identities and retry contracts. ↩
-
Stripe, Idempotent requests. A concrete provider-specific example of parameter matching and idempotency-key retention. ↩
-
AWS Builders' Library, Timeouts, retries, and backoff with jitter. Bounded retries, overload amplification and retry timing. ↩
-
Anthropic, Effective context engineering for AI agents. Selecting and maintaining useful model context. ↩
-
Anthropic, Effective harnesses for long-running agents. Progress artifacts and limitations of compaction alone. ↩
-
Anthropic, Context engineering: memory, compaction, and tool clearing. Distinct context-management primitives. ↩
-
OWASP, Transaction Authorization Cheat Sheet. Binding authorization, invalidation, expiry and final execution checks. ↩ ↩2
-
OpenAI, Understanding prompt injections. Third-party instruction attacks and layered defenses. ↩
-
OWASP, Authorization Cheat Sheet. Least privilege, default denial and permission checks. ↩
-
Model Context Protocol, Authorization Security Considerations. Token audiences, upstream credentials and token-passthrough restrictions. ↩
-
PostgreSQL, Row Security Policies. Policy enforcement and privileged-role exceptions. ↩
-
OWASP, AI Agent Security Cheat Sheet. Agent-specific security boundaries, memory and tool risks. ↩
-
OpenTelemetry, Semantic conventions for GenAI agent spans. Agent-operation instrumentation; the upstream conventions are evolving. ↩
-
Anthropic, Demystifying evals for AI agents. Outcomes, evaluation trials and complementary graders. ↩
-
Cloudflare, Using Agents with Workflows. Responsibilities of agents and durable workflows. ↩
-
Cloudflare, Build a Durable AI Agent. A platform-specific runnable tutorial for a resumable agent loop. ↩
BUILDING AN AI PRODUCT?
Talk to Lofingo about custom autonomous AI agents, enterprise workflow automation, and scalable software systems.
EXPLORE AI SERVICES →