The Resonate Rust SDK is now available to try!

cargo add resonate-sdk

If you write Rust and need code that survives crashes, restarts, and long-running coordination without bolting on a state machine, start here.

What you get is distributed async/await for Rust.

Mark a function with #[resonate::function], call it through ctx.run, and the runtime handles checkpointing, retries, and crash recovery.

You write functions that return Result<T> — no state-machine DSL to author separately.

The programming model matches the TypeScript and Python SDKs, expressed in idiomatic Rust.

To try it: cargo add resonate-sdk and point your project at a Resonate server.

Runnable examples live at github.com/resonatehq under repos prefixed “example-*-rs”.

Reference docs are at docs.resonatehq.io. Questions land best in Discord.

We just open-sourced resonatehq/resonate-skills — 30+ agent skills for building with Resonate. Apache 2.0. TypeScript, Python, Rust, and server operations.

When we say “skill” we mean structured context that teaches coding agents how to reason about and write durable execution code. They follow Anthropic’s Agent Skills convention and work with Claude Code, the Anthropic SDK, Cursor, and any agent that can load Markdown as context.

What’s in the repo:

• Philosophy: This is recommend as the skill every agent should load first. It’s a table of instincts agents have that are wrong when using Resonate.

• Durable Execution: covers the concept itself and a matrix for understanding when you need it.

• Core SDK usage: covering the two execution contexts and debugging.

• Development patterns: one per core pattern.

  • Saga: forward execution with reverse-order compensation.

  • Recursive fan-out: parallel execution via RPCs.

  • Human-in-the-loop: creates a durable suspension point.

  • External system of record: coordinating writes to any external SoR using idempotency keys routed through Resonate’s checkpointing.

  • Durable sleep / scheduled work: timers that survive crashes

• Advanced reasoning: one skill that maps the Distributed Async Await formal spec to SDK patterns.

• Deployments: covers deploying the Resonate server and workers to production.

Skill structure:

Each skill is a directory with a SKILL.md. Frontmatter tells agents when to load it:

---
name: resonate-saga-pattern-typescript

description: Implement saga patterns for distributed transactions with compensation logic. Use when coordinating multi-step processes that need to maintain consistency across failures by unwinding completed steps.

license: Apache-2.0
---

The body is mental models, code patterns, anti-patterns, and decision trees.

To setup fo Claude Code:

cd your-project
mkdir -p .claude/skills
git clone https://github.com/resonatehq/resonate-skills .claude/skills/resonate

To setup for any other agent, load the SKILL.md files as context. No runtime dependency. No package. Plain Markdown.

Invoke explicitly (Use the resonate-saga-pattern-typescript skill to...) or let the agent match on description.

The repo is accepting contributions.

Coverage is expanding as the components evolve.

Resonate is open source. GitHub · Discord · Docs

Distributed Async/Await is the programming model. Agent reliability is the test.

On April 14, I am joining Alexander Gallego on Episode 2 of Redpanda's *Hello, Agent!* podcast to talk about both.

Register: https://www.redpanda.com/events/hello-agent-ep2-distributed-async-await

Most people frame agent reliability as a problem at the edges. Retries. Timeouts. A queue or two. That framing is comfortable. It is also why so many agentic systems fall over the moment they leave the demo.

An agent is a distributed program. Its work spans processes, machines, networks, and time. Once you accept that, the question stops being "how do I make my agent reliable" and starts being "what is the right programming model for distributed computation." That question has a good answer. It is not a new one.

Async/await let us write concurrent code that reads like sequential code. Distributed Async/Await extends the same idea across failure domains. The function suspends. The runtime is responsible for resuming it — same machine, different machine, an hour later, a week later, after a crash, after a deploy. The programming model does not change. The execution model carries the weight.

Possible topics we might cover:

• What "distributed" actually adds to async/await — and why the addition is not cosmetic.

• Designing for reliability when work spans services, queues, and failure domains you do not control.

• Failure handling that does not turn the application into a graveyard of defensive code.

• Why agent reliability and workflow reliability are, in the end, the same problem.

If you have ever written a workflow that looked clean on a whiteboard and then spent a week chasing partial failures in production — this one is for you.

Reliability is not a feature you bolt on. It is a property of the programming model you chose.

April 14. Register early to get first access when the episode drops.

We switched from writing components directly, to doubling down on protocols and specifications.

A formal spec for the distributed async/await protocol. Language-agnostic. Transport-agnostic. Storage-agnostic.

A conformance test suite. Transition tests. Fuzz iterations.

And then we started generating the components.

Not just the SDK. Not just the workers. The server too.

When you need a Resonate server targeting Postgres + HTTP, we generate a server that is Postgres + HTTP. Not a general-purpose binary that happens to support Postgres as one of twelve backends. Every line of code exists because your stack needs it. Nothing else ships.

What does that eliminate?

Generated components don't carry abstraction layers that exist purely for flexibility:

• Fewer state transitions: The promise abstraction alone reduces this to a minimum.

• Fewer network calls: The protocol chatter is incredibly optimized.

• No unused code: Small package sizes mean the components are easy to host, easy to install, and require little resources to run.

And… everything can be serverless.

A generated Resonate server is small and single-purpose enough to run as a serverless function. A Lambda. A Cloudflare Worker. A Supabase Edge Function.

It spins up. It processes work. It spins down.

Your agent workers — built on the Resonate SDK — do the same.

Both the server and the workers run only when there's work to do.

This is great for Agents, because they don't run 24/7. They activate, do work, and go idle. The traditional durable execution model — always-on server clusters, persistence layers, worker processes polling around the clock — means you're paying a fixed infrastructure cost regardless of how much actual work is happening.

With Resonate on serverless, idling costs nothing. And when work does arrive, each invocation does dramatically less overhead per operation because there's no abstraction tax. On pay-per-millisecond infrastructure, less overhead per invocation is directly less money.

Back of the napkin math:

Scale | Resonate. | Leading platforms |
1K executions/day | ~$0/month | ~$25/month |
1M executions/day | $170/month. | $500+/month |
Bursty workloads at extreme scale | <$100/year | ~$80,000/year |

The pattern that agentic workloads produce is intense bursts of orchestration separated by idle periods. The gap between always-on infrastructure and pay-per-use becomes enormous. Combine that with Resonate's reduced overhead per operation and you can get up to ~800x cost differences.

But it can run pretty much anywhere.

Because the protocol is transport and storage agnostic, we can generate components for whatever you're already running. No need to migrate to a whole new platform.

You're running agents on Supabase? We can generate components that run entirely as Edge Functions — durable agent orchestration with zero always-on infrastructure.

You're a larger team with everything on Kafka? We can generate Resonate components that use Kafka for message passing — durable execution on your existing infrastructure.

You need gRPC transport with DynamoDB for state? That's a generation target, not a feature request.

Durable execution has always been the right answer for complex agent orchestration. Multi-step workflows, human-in-the-loop approvals, long-running research tasks, multi-agent coordination — all of this needs crash recovery, checkpointing, and reliable retries.

But the cost of and complexity of the existing durable execution platforms has opened up as many new pains and they solved.

You don't want to have to learn a set of proprietary primitives and spin up $500+/month infrastructure just so your agents can recover from crashes.

When you bring that cost down by 100-800x, durability stops being a premium feature and starts being a default. Every agent workflow gets checkpointing. Every multi-step orchestration gets crash recovery. Every long-running task gets reliable retries. Not because you budgeted for it — because it's so cheap there's no reason not to.

If that sounds interesting — come tell us about your stack in the Resonate Discord.

Durable executions frameworks center their pitches on a mix of vague promises, metaphors, and anecdotes. This post takes a different route: a rigorous definition, independent of any specific framework, along with a conceptual foundation and a minimal working implementation.

Join the discord

Recently, you might have noticed a surge of interest in durable executions; with the transformation of LLMs from turn-based chats to continuously running agents, communicating with each other, orchestrating tool calls, and waiting on humans for confirmation, durable executions are shifting from niche to mainstream.

Every durable execution framework tells a similar story: write normal code, get fault tolerance. Yet none of them define what “normal code” or “fault tolerance” means. Instead they tack on a list of seemingly unrelated claims, such as “tracks your state”, “runs for a long time”, or “sleeps for days”, as though the connections were self-evident1.

The promise of writing normal code is appealing. However, what counts as normal? Some frameworks require developers to use DSLs and structure their applications as workflows and activities—not JSON or YAML encoded directed acyclic graphs (DAGs) but still proprietary programming models with steep learning curves.

The promise of fault tolerance is equally appealing. However, not all failures are created equal. For example, if a payment provider declines a credit card charge, durable executions cannot rectify that situation. If an execution crashes while a payment provider processes a credit card charge, durable executions earn their name.

Resonate’s durable execution framework, Distributed Async Await, is designed around a single principle: simplicity. Not only simple to use, but simple to reason about. That requires a rigorous mental model of durable executions—instead of hand-waving anecdotes and metaphors.

Interruptions

Instead of sleep, crash, fault, or failure, we will reason in terms of interruptions. An interruption consists of two phases:

1. Suspension of an execution

2. Resumption of an execution

A suspension can be internally triggered (e.g., a sleep) or externally triggered (e.g., a crash). Either way, the defining characteristic of durability is the ability to suspend on one process and resume on another process (in practice that may be a green thread, an os thread, an os process, a container, or a machine).

Mitigating Interruptions

Borrowing a technique from programming language theory, we use a simple, idealized case to specify the desired behavior of a complex, realistic case: consider a program P running on a runtime R. In the ideal case, execution proceeds without interruptions
-I. The program runs from start to finish, producing some observable behavior, a trace T.

P/(R−I) = T

In words: Program P running on runtime R without the possibility of interruptions
-I produces trace T

Now introduce interruptions. Execution may suspend and resume, potentially on a different process. We want the same observable behavior T as if no interruption occurred. In other words, we want interruption tolerant executions.

How do we achieve this? There are two approaches, two knobs we can dial.

Approach 1: Interruption-Aware Definitions

Modify the program. Transform P into P′ such that:

P/(R−I) ≣ P′/(R+I)

In words: Program P running on runtime R without the possibility of interruptions
-I produces an equivalent trace to a modified program P` running on the same runtime R with the possibility of interruptions +I

The developer explicitly handles interruptions. They add snapshotting, checkpointing, and recovery logic. The program becomes aware of and responsible for interruption handling, riddled with interrupt specific concerns.

Approach 2: Interruption-Agnostic Definitions

Modify the runtime. Transform R into R′ such that:

P/(R−I) ≣ P/(R′+I)

In words: Program P running on runtime R without the possibility of interruptions
-I produces an equivalent trace to the same program P running on a modified runtime R’ with the possibility of interruptions +I

The programmer writes the same program P. The runtime handles interruptions transparently. The program remains agnostic to whether interruptions occur.

Durable Executions

We can now formulate a precise definition: Durable Executions are Interruption-Agnostic Definitions of Functions that result in Interruption-Tolerant Executions of Functions.

The definition has two parts:

1. Interruption-Agnostic Definition: The function definition ignores the possibility of interruptions.

2. Interruption-Tolerant Execution: The function execution compensates for the occurrence of interruptions.

According to this definition, “normal code” refers to function definitions that are interruption-agnostic i.e. do not account for the possibility of interruptions while “fault-tolerant” refers to function executions that are interruption-tolerant i.e. compensate for the occurrence of interruptions and show the same observable behavior whether interruptions occur or not.

A minimal implementation

Can we implement durable executions without rewriting compilers and runtimes from scratch? Can we implement them with a language-specific SDK? While this requires some syntactic compromises (e.g., the use of yield instead of await or the use of decorators), we can preserve normal code semantics: functions calling functions calling functions, with the usual control structures such as if, for, while, try-catch.

Let’s try.

A note: our durable executions won’t actually be durable. We’re focusing on execution mechanics, not persistence. The runtimes below keep state in memory to illustrate how execution unfolds, not how storage works.

The Program

Consider a simple program of nested function calls. A function is implemented as a coroutine (a.k.a generator):

function* foo(): Generator<any, number, any> {

  let a = yield call(bar, 21);

  let b = yield call(bar, 42);

  return a + b;

}

function* bar(arg: number): Generator<any, number, any> {

  let a = yield call(baz, arg);

  let b = yield call(baz, arg);

  return a + b;

}

function* baz(arg: number): Generator<any, number, any> {
  return 2 * arg;
}

yield call(function, arguments) marks function calls: foo calls bar twice, bar calls baz twice, and baz performs a simple computation.

The expected result is:

• baz(21) returns 42, so bar(21) returns 42 + 42 = 84

• baz(42) returns 84, so bar(42) returns 84 + 84 = 168

• foo() returns 84 + 168 = 252

The call helper creates a descriptor for the function call:

class Call {
  constructor(
    public func: Function,
    public args: any[],
  ) {}
}

function call(func: Function, ...args: any[]) {
  return new Call(func, args);
}

Two Runtimes, Same Result

We implement two different runtimes, one that yields correct and complete results only without interruptions and one that yields correct and complete results with interruptions.

Resume Runtime

The resume runtime executes coroutines in a single pass. When a Call effect is yielded, the runtime executes the callee, passes the result to the caller, and resumes:

function resume(func: Function, args: any[]): any {
  let coroutine = func(...args);
  let value: any = undefined;

  while (true) {
    let result = coroutine.next(value);

    if (result.done) {
      return result.value;
    }

    const call = result.value as Call;
    value = execute(call.func, call.args);
  }
}

This runtime implements the easy case: uninterrupted executions. The run function simply invokes resume, which executes in one turn:

function run(func: Function, ...args: any[]): any {
  return resume(func, args);
}

Running foo yields:

console.log(run(foo)) => 252

Restart Runtime

The restart runtime executes coroutines in many passes. When a Call effect is yielded, the runtime checks the cache and either:

• cache miss
  Adds the call to the cache and interrupts

• cache hit, pending
  Executes the callee, adds the result to the cache, and interrupts

• cache hit, resolved
  Passes the cached result to the caller and resumes

type Result = 
  { status: “pending” } 
| { status: “resolved”; value: any };

function restrart(
  func: Function, 
  args: any[], 
  pid: string, 
  env: Record<string, Result>
): any {
  const coroutine = func(...args);
  let value: any = undefined;
  let count = 0;

  while (true) {
    const result = coroutine.next(value);

    if (result.done) {
      return result.value;
    }

    const call = result.value as Call;

    // Generate a deterministic call site id

    const cid = `${pid}.${count++}`;

    if (!(cid in env)) {
      env[cid] = { status: “pending” };
      throw new Error(”Interrupt”);
    }

    const cache = env[cid];

    if (cache.status === “pending”) {
      const res = execute(call.func, call.args, cid, env);
      env[cid] = { status: “resolved”, value: res };
      throw new Error(”Interrupt”);
    }

    value = cache.value;
  }
}

A note on identity and determinism: The caller provides a root ID (e.g., 0), and the runtime generates child IDs based on position in the call tree (e.g., 0.0, 0.1, 0.1.0). On restart, these IDs map to cached results. This requires executions to be deterministic with respect to the call tree.

A note on idempotence: If an execution is interrupted after the function executes but before the result is recorded, the function will be executed again. Therefore, in practice, functions must be idempotent.

This runtime implements the hard case: aggressively interrupted executions. The run function loops until completion, catching interrupts and restarting:

function run(id: string, func: Function, ...args: any[]): any {
  const env: Record<string, Result> = {};
  while (true) {
    try {
      return restart(foo, args, id, env);
    } catch (e) {
      if (e.message === “Interrupt”) continue;
      throw e;
    }
  }
}

However, despite constant interruptions, the final result is identical to uninterrupted execution:

console.log(run("0", foo)) => 252

The program is unchanged. The runtime compensates for interruptions. This is exactly what our definition promised: interruption-agnostic definitions yielding interruption-tolerant executions.

Notice what we did not do:

• We did not use a DSL or workflows and activities

• We did not write checkpointing code

• We did not write serialization code

• We did not write recovery code

Did we write normal code? The syntax is unfamiliar, yield call(bar, 21) instead of bar(21), coroutines instead of traditional functions. However, the semantics are unchanged. We didn’t need to invent a proprietary programming model.

Conclusion

Durable executions are not fuzzy but a well-defined concept: Interruption-Agnostic Definitions of Functions that result in Interruption-Tolerant Executions of Functions—made possible by reliable restarts and memoization.

So the next time someone tells you that durable executions let you “write normal code and get fault tolerance,” you know exactly what that means.

Try Resonate

Get started in minutes and explore our examples: a durable deep research agent that scales to hundreds of subagents in under 110 lines, or a durable countdown that runs for days, weeks, or months in under 30 lines of code. Run them on your laptop, your cluster, or even on serverless platforms—the code stays the same.

If you want to learn more about the mechanics of Resonate, check out my Systems Distributed ‘25 presentation, where I walk through Resonate’s programming model and execution model.

I wanted to see what it would take to deploy a remote Resonate Server—one that exposes public endpoints my local apps could connect to.

I didn’t have a specific use case in mind. I just wanted to run a few durable applications against a real, remote Resonate Server—not one on my local network. I’d been experimenting with DigitalOcean Droplets, so I decided to use one for the job.

This turned out to be a really fun project—especially if you enjoy working directly from the command line. If you prefer a graphical configuration screen, this might not be your ideal path. But if you like understanding how every piece fits together, this approach will feel satisfying.

The networking setup gave me the most trouble (as it often does), but working through that taught me a lot—and that’s exactly why I’m sharing this tutorial. My trial and error should save you several hours if you decide to try the same thing.

In the end, it worked beautifully: I connected multiple local apps to my new remote Resonate Server, and everything ran smoothly and durably.

👉 If you want to deploy your own Resonate Server on a DigitalOcean Droplet, follow this step-by-step guide, “Run a Resonate Server on a DigitalOcean Droplet”.

Let’s talk about developing modern distributed applications! 👻

On Halloween at 10 AM PT, Resonate CEO and distributed‑systems engineer Dominik Tornow will be hosting an Ask‑Me‑Anything.

Bring your darkest questions about building distributed protocols, DST (Deterministic Simulation Testing, Durable Execution or anything on your mind. Dominik’s specialty is making complex systems dead simple — and he’ll be answering your questions live.

Resonate is sponsoring this AMA because we believe developers deserve a quality‑of‑life upgrade. Our platform replaces duct‑taped queues and schedulers with a single, developer‑friendly model . We offer async/await across your entire application, so you can build reliable, scalable systems that just work . In Dominik’s words, it’s about turning the hardest parts of distributed systems into a few lines of code.

So get your questions ready and join us! 🕸️ Bring your curiosity (and optionally a costume) — we’ll be in the Resonate Discord answering anything from the basics of Resonate’s durable async/await to your own haunted debugging stories. Can’t wait to see you there!

📅 When: Friday, Oct 31 at 10 AM PT

📍 Where: Resonate Discord https://resonatehq.io/discord

🎭 Theme: Halloween AMA – costumes optional 🎃

See you on Halloween!

Chapter 2 drops

Chapter 2 of Systems Engineering for Agentic Applications published earlier this month. begins breaking down everything you need to know about Building an Agent.

If you missed Chapter 1, about going “From AI to Agentic Applications”, the entire book is free and new chapters are published at the beginning of the month.

You can subscribe to updates on Dominik’s Agentic Applications Substack.

Calling on contributors

Come help implement the next round of features across the Resonate system.

1. Pluggable auth system

Add modular auth for SDK ↔ Server communication. Enabling secure, real-world production use for a wide range auth preferences.

2. Message transport plugins

Write connectors for Kafka, RabbitMQ, Redis, Pub/Sub, etc - Making Resonate compatible with any architecture.

Community member asgr (Muhammad Asghar Ali) has been heroically driving the design of that forward giving the ecosystem an SQS plugin and Nats.io plugin on top of the existing ones.

3. Privacy features

Start by adding encode/decode apis with pluggable encoding algorithms to ensure developers have a quick and easy way to implement encryption for their user’s data while it’s inside Resonate promise stores and traveling across message transports. This is needed for real life use cases.

4. Observability (metrics, logging, tracing, etc…)

Design visibility into everything: execution paths, retries, performance. Great observability increases the quality-of-life for a developer.

Resonate Server v0.7.18 release

Highlights include a new configuration that manages the lifecycle of tcp connections for a smoother google cloud run experience.

→ Resonate Server v0.7.18 release notes

Resonate TypeScript SDK v0.8.3 release

Highlights include a new "until” configuration for the sleep API (i.e. “sleep until” functionality).

→ Resonate TypeScript SDK v0.8.3 release notes

Resonate Python SDK v0.6.7 release

Highlights include a few minor bug fixes.

→ Resonate TypeScript SDK v.0.6.7 release notes

Resonate offers developers a dead simple holistic programming model for building distributed applications.

Resonate Status Update: Fall 2025

TypeScript SDK v0.8.0 release.

We would be lying if we said “we weren’t impressed with ourselves” with this release. It polishes up a complete refactor of the SDK to align it with Resonate’s revised core API:

• Run (Synchronous, Local)

• Begin Run (Asynchronous, Local)

• RPC (Synchronous, Remote)

• Begin RPC (Asynchronous, Remote)

If you love working in TypeScript - you should try out this SDK.

Python SDK v.0.6.4

Thanks to the efforts of Karn Wong, their benchmark tests uncovered a much needed change to improve compatibility with async applications, which was released in v.0.6.4.

Resonate Server v.0.7.15 release

The latest Resonate Server release includes an SQS plugin that enables Async RPC, while using SQS as the message transport. All appreciation, applause, and admiration may be forwarded to Muhammad Asghar Ali for their heroic efforts in bringing this functionality to fruition!

Getting started example apps

The team was excited to launch this curated set of example applications designed to help developers get started with Resonate within the context of a familiar use case.

Hacktoberfest

Resonate’s open source component repos are available to Hacktoberfest participants.

• Resonate Server

• Resonate Python SDK

• Resonate TypeScript SDK

The team made a solid effort to update issue descriptions and tags to be more helpful to outside contributors.

A lot of amazing things are made from someone else’s example and we hope you will make amazing things from these ones. Set within the context of specific use cases, Resonate’s programming model is put to the test in minimalistic example applications.

Load balanced worker fleet

Available in Python, TypeScript.

Have you ever needed to balance load across a fleet of Workers? Resonate lets you run as many workers as you need. Create distinct worker groups and route to them within a simple API.

Try the load balancing example!

Kafka batch processing worker

Available in Python.

Have you ever needed a worker for a Kafka Topic and had to solve for head-of-line-blocking, retries, and recovery? Resonate can help with that. A real treat is using Redpanda as a drop-in replacement for Kafka, making it easy to observe the data movement.

Try the Kafka worker example!

Waiting on a Human-In-The-Loop

Available in Python, TypeScript.

How often have you tried to pause a multi-step workflow to wait for someone to click a link in an email? See how to solve for this with Resonate, suspending a function execution for an indefinite amount of time.

Try the Human-In-The-Loop example!

Website Summarization Agent

Available in Python.

Want a little taste of everything? This example, while remaining surprising minimal packs in a workflow decked out with two failure prone steps, a human-in-the-loop, a non-retriable error, and result caching by ID. Thanks to Resonate’s programming model, what would normally take a lot of custom code to handle, these important distributed systems issues are boiled down to simple API calls.

Try the website summarization agent example!

Indefinite Durable Sleep

Available in Python, TypeScript.

Tired of using cron jobs to awaken long sleeping / long running business processes? Check out this one line of code that puts your workflow function to sleep for days, weeks, or years while surviving the lifetime of the processes in which it executes!

Try the durable sleep example!

Find more examples like these in the resonatehq-examples Github Org.

Resonate Workers make extending durability, to span between that RedPanda Topics, dead simple.

Check out the app!

Our latest example application showcases what a RedPanda + Resonate pipeline might look like in the context of a "batch record deletion" use case.

RedPanda is an amazingly approachable Kafka replacement that provides durable message topics (queues).

Resonate provides:

• Out-of-the-box concurrent message processing to solve for head-of-line blocking issues.

• Out-of-the-box function execution retries to solve for transient errors in message processing.

• Out-of-the-box recovery and resumption from processing node crash failures.

• Dead simple programming model.

Check out the example application to see for yourself!

Check out the app!

If you are building an MCP server then you are also building a distributed system.

The square hole

The main issue with MCP, and all LLM tool calling functionality at present, is that it is synchronous with no built-in mechanisms for handling failure.

Even though MCP standardizes a tool calling convention - it ignores all the other issues that arise from building a distributed system and as the developer you are burdened with figuring out:

• Supervision, such as timeouts, to detect issues

• Retry logic for application level errors

• Deduplication and/or idempotency guarantees

• Recovery for process crashes

But… what if you weren’t?

Pinky promise

The MCP Server Quickstart guide uses weather forecasting as an example use case. Let’s flip it around and use historic weather data gathering as our example, a use case which could take a much longer time and better reflect the need for a background job.

To enable asynchronous behavior, instead of a single tool such as get_weather_data, we will create three tools:

• start_gathering

• probe_status

• await_result

And here is the key — The start_gathering tool, instead of blocking on the result of the data gather job, kicks off a background job and returns a promise ID — the secret ingredient to solving for our distributed system problems.

Let’s have a look at the code that makes it possible with Resonate:

In the previous code example, the job_name doubles as the promise ID. We don’t need to burden our LLM with learning about promises and concurrency. We just need to make it clear that this tool starts a background job, and returns the name of the job which can be used to get the status or result.

How is this possible?

Decorate with Resonate

Resonate breaks the code up into two worlds:

• Ephemeral World

• Durable World

MCP lives in the Ephemeral World, if you were just using MCP and your server crashed mid-tool usage, then you are on your own in figuring out how to handle that.

Resonate enables MCP tools to transition to the Durable World, and as soon as you transition to the Durable World you are covered — Resonate gives you everything you need to handle idempotency, application errors, process failures, distribution, and concurrency.

Decorate the start_gathering() function with @mcp.tool, registering it as a tool with MCP. This function will execute in the Ephemeral World.

Decorate the weather_data() function with @resonate.register, registering the function with Resonate. This function will execute in the Durable World.

Within the start_gathering() function we call weather_data.run(), which transitions the call chain from the Ephemeral World to the Durable World. This call returns a handle to the invocation. You could wait right there for the result using the handle, but instead we will return the promise ID (job name) we used to invoke the weather_data() function.

The promise ID (job name) is also an idempotency key, enabling us to rejoin that invocation from almost anywhere. This is because Resonate promises are Durable Promises. That job name will always give you the same handle, and once resolved, will always give you the same result, instantly.

Are we there yet?

In the previous example, to keep things simple, we are accepting a single location at a time. However, you could alter it to kick off a set of jobs for a set of locations.

But to hammer in the value of Resonate Durable Promises, we want to look at checking the status and awaiting the results in sets. So that you can do something like this with your LLM:

The await_result() and probe_status() tools take a set of promise IDs (job names).

The probe_status() tool just checks if the data is ready by using the promise ID to get the handle of the invocation and calling .done() to check if the promise is Resolved yet.

So you can do something like this:

The await_result() tool will actually block and await on the results of the background jobs by calling .result() on the handle.

And this turns what was a previously a completely synchronous and fragile tool interaction, into a durable and asynchronous data gathering and analysis experience.

The best of both worlds

Integrating Resonate into a MCP server preserves standardized tool calling while completely transforming your application into a Durable Distributed System.

You don’t need to teach the model about distributed systems.

You don’t need to wrap every tool in scaffolding just to handle timeouts or retries.

You don’t need to worry if your server crashes halfway through a job.

Check out the weather data agent tool example application to see for yourself.

You can apply that quote to our species obsession with the advancement of AI in general. However, here I am applying it to the evolution of LLM tool calling capabilities.

Mad tool skills, bruh

LLMs are now trained to provide structured data responses when they “want” to make use of a tool.

For example, if I prompt a model that has been trained to make tool calls to gather weather data for Parkland County, Alberta (that’s where I live) for July 2023, the model will know to format its response to my question with something like this:

The chat client, or application that interacts with the LLM, is responsible for looking at the output, detecting the structured response, parsing it, and then invoking the “get_weather_data” function with the arguments.

The expectation is that the result of the tool function is then passed back to the LLM as part of the conversation. That is, the LLM, as part of its training, expects the next input to be a result from the tool call.

Remember, remember

Remember that an LLM by itself does not have memory. It relies on the entire conversation thread (also called context) to be provided on each and every interaction, with each user input, LLM response, or tool result appended to the context/conversation thread. In other words, the interaction is turn based, and on each turn the entire history is fed back into the LLM.

Note, the big players like Open AI and Anthropic have added RAG-like memory to their chat clients so that there is an appearance of memory across different context windows (they persist the conversation threads and search them for information, passing it to the LLM as part of other conversation threads). But fundamentally they are working around the memory constraint described above.

Sync, sync, baby

Currently interactions with tools are synchronous. And it is hard to imagine it will be anything other than that.

If gathering weather data takes 8 hours, perhaps for a very large area over a longer time period, then that context thread needs to wait 8 hours for the tool result before the user can add additional input and continue interacting with the LLM within that thread.

As a workaround, Open AI has managed to train some of its models to invoke a tool multiple times from a single model response. That is, if I were to ask their model to “gather weather data for July 2022 and July 2023 for Parkland County, Alberta” it would know how to return a response like this:

I think you might call this the “fork / join technique”.

So, if data gathering for each location takes 8 hours, instead of taking 16 hours for back-to-back tool calls with a prompt between them, it would only take 8 hours before you could move forward with the thread.

But, again, the interaction is synchronous and you can’t move forward with that specific context/conversation thread until there is a result from the interaction with the tool calls. The LLM would throw an exception otherwise.

But what about MCP?

MCP (Model Context Protocol) standardizes the approach. That is, MCP creates a standard by which all LLMs can be trained to structure their output to connect to and invoke tools that exist outside of a chat client or application. Which is cool.

But MCP doubles down on this synchronous behaviour. It has no built-in mechanisms for handling the distribution that it facilitates. It creates TCP connections that need to be maintained throughout the entire tool interaction. And if the tool errors, the MCP Server crashes, or the connection between the chat client / application and the tool fails, there is no built-in means by which to mitigate that failure or those errors.

Even though MCP standardizes a tool calling convention, it ignores all the other issues that arise from building a distributed system and as the developer you are now burdened with figuring out:

• Supervision, such as timeouts, to detect issues

• Retry logic for application level errors

• Deduplication and/or idempotency guarantees

• Recovery for process crashes

Congrats, you are now engineering a reliable distributed system.

Strongest of hype cycle

This situation continues to exacerbate a problem where application developers, and in this case autonomous application developers, are forced to be distributed systems engineers.

So, while AI speeds things up in theory, there will still be a delay in pushing reliable autonomous applications to production, because the developers still need to wrestle with the same issues they have been wrestling with since an applications started spanning more than one process almost three decades ago.

Unless…

Unless Resonate has something up its sleeve that could take all that pain away?

Stay tuned to find out!

Worker

A Worker, also known as an Application Node, is a single process with an instance of Resonate. An application can have between 1 and n unique Workers.

Group

A Group is a set of Workers that share a Group ID.

Workers in a Group should be identical instances of each other. That is — a Group of Workers are processes with identical code bases.

An application can have between 1 and n unique Groups.

Fleet

A Fleet is the application’s full set of Workers and Groups.

A Fleet can have a between 1 and n unique Workers and 1 and n unique Groups.

This blog post introduces Distributed Async Await, a dead simple programming model for concurrent and distributed applications, designed to provide a delightful developer experience.

This is the first in a series of three blog post:

• Introduction

• The Protocol Stack

• The Programming Model

The Challenge: Concurrency and Distribution

Today’s applications operate in distributed environments, handling large numbers of concurrent executions across multiple processes. In short, today’s applications are characterized by concurrency and distribution.

Concurrency and distribution present two challenges:

• Concurrency

  Concurrency introduces non-deterministic partial order, where the sequence of operations is unpredictable across components. In other words, we do not know what happens next.

• Distribution

  Distribution introduces non-deterministic partial failure, where the occurrence of failure is unpredictable across components. In other words, we do not know what fails next.

To mitigate the effects of concurrency, we need distributed coordination and to mitigate the effects of distribution we need distributed recovery:

• Distributed Coordination

  Distributed Coordination is the ability of a system to mitigate partial order by synchronizing between executions, potentially running on different processes.

• Distributed Recovery

  Distributed Recovery is the ability of a system to mitigate partial failure by resuming an execution in case of failure, potentially on a different process.

Concurrent programming models such as async await have simplified the challenges of concurrency within a single process. However, no comparable concurrent and distributed programming models have emerged.

Developers have to address the complexity manually, cobbling together snapshotting, checkpointing, polling, or retrying. This resulting code that is hard to create, hard to update, brittle, buggy, and obscures its very reason to exist: its business logic.

Example: The Countdown

Consider a simple use case, a countdown timer. We initiate a countdown with a phone number, a count, and a frequency. The app will send a text message to the provided phone number in the provided frequency, decreasing the count by one each time, until we reach zero.

To make the challenge more obvious, we execute the countdown on a Function-as-a-Service platform such as AWS Lambda. That introduces two problems:

• Time Limit

  A lambda function can run for a maximum of 15 minutes before being forcibly terminated.

• Cost

  Even if your countdown fits within the time limit, you are billed for the entire runtime, even though the function is mostly idle.

For example, a countdown starting at 10, with a frequency of 15 minutes will take 135 minutes, or 2 hours and 15 minutes to complete. Performing a back-of-the-envelope calculation, assuming sending a message takes 500ms, the execution is active for 5 seconds and asleep for 2 hours, 14 minutes and 55 seconds. First, we are exceeding the 15 min time limit and second we would be paying for 2 hours and 15 minutes instead of 5 seconds.

So, instead of simply suspending and subsequently resuming the execution, due to time limit and costs, we have to suspend the execution, terminate the process, subsequently initialize another process, and resume the execution－until today, manually.

Solution #1: Event Driven Architecture

One possible solution is an event-driven architecture: Instead of trying to maintain an active execution of countdown, we split the countdown into multiple (recursive) callbacks, each triggered after the delay:

// Bound to http://example.com/countdown
faas.handler('countdown', async (phone, count, delay) => { 
  // send the current countdown
  send(phone, count);
  // schedule another invocation for the future
  if(count => 0) {
    await cron.when({ 
      post: 'http://example.com/countdown', 
      data: [phone, count - 1, delay]
    }, delay);
  }
});

This approach comes at a cost: We need to manage the state of the execution, that is, we need to manage the instruction pointer, the stack, and the heap, a consequence of event driven architectures known as stack ripping or callback hell－the code looks and feels fragmented across space and time.

The obfuscated control flow is not our only issue tho. Like most ad-hoc solutions, we face the most difficult problems in case of failure.

countdown lacks any notion of principled failure handling semantics:

• What happens if the invocation of send fails? Will the invocation be retried? Who is responsible for the retry? Will the message be sent again?

• What happens if the invocation of cron fails? Will the invocation be retried? Who is responsible for the retry? Will the next countdown be scheduled twice?

• What happens if cron itself fails? Will cron retry the http call? Will the next countdown be invoked twice?

Maybe basing a long running execution such as countdown on ephemeral http requests is not the best call. Maybe we should switch to durable queues?

To make matters worse, countdown lacks any notion of debugability and observability:

• How do we learn that a countdown has failed mid execution?

• How do we learn what countdowns are currently executing?

• How do we diagnose a problem? Where do we look?

• How do you cancel an ongoing countdown?

Maybe not having a manifestation of a long running execution except some job in cron or some message in a message queue is not the best call either. Maybe we should add a database?

The result? Dread!

We face an endless list of questions that we must answer one by one, analyzing the behavior of the individual components and how they interact, only to start from the very beginning after we make the slightest change.

The result? The code is neither easy to write nor easy to read and only indirectly, in a roundabout, obfuscated way, reflects our business logic. Reasoning about failure? Confusion. Adding new functionality? Frustration.

Even if we acknowledge that most of us can handle the complexity, I would argue none of us wants to. This is not a fun challenge, this is joyless toil－a dreadful developer experience.

Solution #2: Distributed Async Await

Could we write code as if we suspended and subsequently resumed?! On this or a different process, with or without failure?! Could we enjoy a delightful developer experience and achieve the illusion of continuity across space and time?!

async countdown (context, phone, count, delay) { 
  for (let i = count; i > 0; i--) {
    // 1. Locally invoke and await a function send
    await async send(phone, i);
    // 2. Remotely invoke and await a timer
    await async wait(delay);
  }
}

Distributed Async Await is available as SDKs for different programming languages like typescript, javascript, or python－the example abstractly showcases the core primitives, async and await.

This code is easy to write, easy to read, and directly reflects our business logic. No cron jobs, no message queues, no databases.

Just a simple function.

The result? Delight!

Resonate’s Distributed Async Await is a programming model with language-integrated distributed coordination and distributed recovery, enabling developers to express complex logic with simple code.

Distributed Async Await is built on two abstractions, durable functions and durable promises:

• Durable Functions

  Functions that run to completion, even in the presence of interruptions.

• Durable Promises

  Promises that coordinate between executions on the same or different processes.

Distributed Async Await is a dead simple programming model that enables developers to build applications that are

• simple to understand

• simple to develop

• simple to operate

The next blog posts will explore the mechanics of Distribute Async Await to gain an accurate and concise mental model

During a discussion about determinism and Durable Execution with a super interesting point was made.

With a log-based Durable Execution system, like Temporal and Restate, your “workflow code” needs to be deterministic to make the system happy. With an object set-based system, like Resonate, your “workflow code” needs to be deterministic to make your use-case happy.

I spent several years working on Temporal’s documentation. One of the challenges was making “getting started with Temporal” content approachable, but also guide developers around the pitfall of non-determinism in their workflows.

From my perspective, this was difficult because, as a developer, to craft workflows that could actually recover from crash failures, you inevitably needed to understand how replay worked. The catch-22 was that no one really wanted to spend the time to learn a deep technical concept just to adopt a new technology. And yet it became fairly necessary to address early on in a developer’s journey.

All that is to say, I took some deep dives into the inner workings of Temporal’s replay algorithm to try and understand where that line might be — the balance between learning replay and getting yourself started.

I am still not sure I can confidently say there is a one-size fits all approach there. Distributed Systems are complex, and every developer is on a different journey with a different set of prejudices towards patterns they like and don’t like.

However, I did walk away with a real interest in Durable Execution systems and confidence that comes from having a solid mental model. So, it has been incredibly fun to dive deep into how Resonate works and compare it with Temporal and others.

The insight about where deterministic constraints comes from (to make the system happy or to your use case happy) comes from exploring the method by which each system makes replay possible — replay being the means by which a function can recover after the process it was executing in crashes.

Log-based systems

Temporal and Restate both maintain a dedicated and sequenced log of steps per workflow invocation. Temporal calls it an Event History, Restate calls it a Journal.

When a workflow function needs to recover, it is re-executed in a new process and each step of the execution is compared to its log. To prevent unwanted side effects, values are pulled from the log for steps that are tracked there. This is done until the e execution progresses past the last step in the log.

Example log of a completed workflow function:

Workflow: foo started {args:[]}
Step: bar invoked {args: []}
Step: bar returned {result: 1}
Step: baz invoked {args: [1]}
Step: baz returned {result: 2}
Step: qux invoked {args: [2]}
Step: qux returned {result: 3}
Workflow: foo completed {result: 3)

For this to work, you have to separate the workflow code from steps that introduce any randomness, such as interactions with APIs or file systems, or generating random numbers. Because during a replay, the steps in the execution must match the steps in the log. If there is a mismatch, the system can not guarantee recovery.

Here’s a practical example, let’s say you add a random number generator to your workflow function, giving a 50% chance of invoking baz().

Example pseudo-code:

@workflow
func foo(ctx) {
    result = await ctx.step(bar)
    if( random(0-100) > 50 ) {
        result = await ctx.step(baz, result)
    }
    result = await ctx.step(qux, result)

    return result
}

// activity
func bar() {
   // ...
   return 1
}

// activity
func baz(arg) {
   // ...
   return arg + 1
}

func qux(arg) {
    // ...
    return arg + 1
}

Let’s say that on the first execution attempt baz() is invoked and returns, but just after that the process crashes. The log might look like this:

Workflow: foo started {args:[]}
Step: bar invoked {args: []}
Step: bar returned {result: 1}
Step: baz invoked {args: [1]}
Step: baz returned {result: 2}

Let’s say another process picks up the task to execute the workflow function and executes it for the second time. However, on the second execution baz() is not invoked. When the execution reaches the invocation of qux() there is a mismatch between the current execution and the log and now the system doesn’t know what to do.

Workflow: foo started {args:[]}
Step: bar invoked {args: []}
Step: bar returned {result: 1}
Step: qux invoked {args: [1]} !DETERMINISTIC REPLAY ERROR

It doesn’t matter if your use case allows for non-determinism or not. The system can’t have that happen, or it can’t recover your workflow execution.

Object set-based systems

Resonate also replays function executions to recover after a process crashes.

However, instead of having a dedicated log and specifying a specific function for which the log applies, Resonate uses un-sequenced promises.

During the Replay of a function execution values are pulled from promises. When the execution reaches a promise for which there isn’t a result yet, it invokes the step (in most cases that’s a function) and waits for the value, indicating it has progressed further than the previous execution.

Consider the same situation as previously where there is a 50% chance of invoking baz(), and on the first execution baz() is invoked and returns.

The following promises would exist:

PromiseFoo: {Status: Pending, Result: Null}

PromiseBar: {Status: Resolved, Result: 1}

PromiseBaz: {Status: Resolved, Result: 2}

Again, assuming the process crashed after baz() returned, foo() would re-execute in a new process. The value of bar() would come from the previously completed promise to prevent duplicate side effects. However, in this case, the system would not prevent the execution from continuing if baz() is skipped.

The execution would just progress to qux() because there is no log sequence dictating that baz() should be the next step. The resulting promise states would look like this:

PromiseFoo: {Status: Resolved, Result: 2}

PromiseBar: {Status: Resolved, Result: 1}

PromiseQux: {Status: Resolved, Result: 2}

Using this replay method, you don’t need to separate functions by which ones have to be deterministic and which ones can engage in randomness.

However, on the first execution baz() did execute to completion, and whether that is bad or not is up to your use case. Something undesirable could happen if you are not aware of how replay works.

Understanding replay

The theme that I see, is that to be successful with any Durable Execution platform, a developer will inevitably need to understand how replay works to recover executions after process crashes. There doesn’t seem to be a way around that.

A log-based approach could almost be considered a “guard-rail” approach, because a log-based system doesn’t allow progress past the point of contention, and in many cases enables a fix to be deployed to solve for it. The upfront learning curve includes both the system’s replay semantics as well as the primitives you need to use to work within the system, such as “workflows”, “events”, etc, thus warranting the “guard rails”.

An object set-based approach takes the guard rails off. So, it is important to understand replay and how it could affect your use case pretty early on. But since this approach builds on existing and well known primitives (functions and promises), the learning could be far less steep, leaving plenty of room for a developer to wrap their head around the concept without too much struggle.

There are many more aspects of these systems to compare — even on the topic of deterministic constraints, this barely scratches the surface. But for whatever reason I found the insight between the two types of systems incredibly compelling and worth thinking about.

I hope you enjoyed thinking about it too!

When we started Resonate in June 2023 we had one goal: To solve complex problems with simple code, so developers can focus on creating value instead of fighting frustration.

Complexity has outpaced comprehension

Modern applications are complex. They are concurrent and distributed, characterized by a high degree of non-determinism (an elegant way of saying utter chaos), and stall or fail in countless ways that are hard to understand and even harder to address.

In practice, developers are forced to choose between two painful extremes:

• Build a distributed application from scratch, wiring together databases, queues, and timers in an intangible code base we then call “event driven architecture”.

• Submit to a wave of orchestration engines, with proprietary programming models, built on proprietary protocols, and running on a proprietary centralized platform.

Neither path feels good. There has to be a better way.

A Dead Simple distributed programming model

Resonate is not building another orchestration engine, platform, framework, or library. We are engineering a new programming model: Distributed Async Await

Just like async await empowers developers to build scalable and reliable concurrent applications without having to focus on the pitfalls of concurrency (deadlocks and race conditions, anyone?), Distributed Async Await brings the same benefits to building scalable and reliable distributed applications.

Write normal code in Python, Typescript, or Javascript, but behind the scenes, Distributed Async Await handles the complexities of distributed applications.

Coordination and recovery are integral to the programming model itself. Dependency management between executions, failure detection and mitigation are all built in.

Distributed Async Await makes building distributed applications as simple as writing a function. Focus on what your application does, not how your application coordinates or recovers. We have you covered.

Build to empower, not to entrap

Many solutions open source their code but keep their core proprietary: A proprietary programming model, executing on a proprietary protocol, running on a centralized platform. The entire stack is a trap.

Resonate hits different. When you adopt Distributed Async Await, you’re not adopting a proprietary tech stack. You’re adopting an open programming model－async await－extended for distributed systems. The protocols powering that extension are themselves based on open specifications.

Both the programming model and the protocols are designed to be dead simple, to thrive in a decentralized environment, and to be implementable by anyone across any language and technology—not just by one vendor and their platform.

A Bold Vision, a Bold Foundation

We’re proud to partner with a group of extraordinary investors who share in our vision to simplify distributed application development:

Amplify , Firestreak, Industry Ventures, Moth Fund, NotVCs, PebbleBed, Sunflower Capital

With $3 million in seed funding, Resonate HQ is on a mission to make writing complex distributed applications as easy as writing a for loop ✌️

Want to build autonomous, long-running agents powered by OpenAI?

You can with Resonate's Distributed Async Await.

Get started with this Reminder Assistant.

Built using a minimal agent loop, the assistant can wait hours, days, or weeks without relying on cron jobs, queues, or databases－thanks to Resonate's durable execution model.

The reminder assistant enables users to schedule reminders through natural language:

Remind me first thing in the morning, bright and early, to check out https://resonatehq.io

The Reminder Assistant is implemented as an agent loop using Resonate's Distributed Async Await. Distributed Async Await enables durable suspension and resumption of long-lived executions.

With Distributed Async Await, the scheduling logic is Dead Simple:

• The schedule function yields (i.e., sleeps) until the target time

• The system automatically resumes when the timer expires

• The AI immediately sends the reminder afterward

Stateful Agent = Stateless Agent + Stateful Execution

The AI Agent does not keep track of state or time, both of which are handled by Resonate: The agent is part of a durable runtime that can wait hours, days, or weeks and still complete the task. The runtime wakes the agent when it's time to act.

This eliminates the need for traditional scheduling infrastructure like cron jobs, message queues, or databases.

The assistant leverages OpenAI’s GPT-4 model with structured function calls.

System Prompt:

You are a helpful assistant that schedules reminders using three tools: 'schedule', 'reminder', and 'current_time'.

Users may express reminder requests ambiguously (e.g., "tomorrow morning, bright and early"). Use your best judgment to interpret such time expressions. Do not ask for clarification.

When a user asks for a reminder:

1. Call the 'schedule' tool with the desired UTC timestamp. This pauses the conversation until that time.

2. Once the system resumes and sends you the next message, the scheduled time has arrived.

3. At that point, immediately call the `reminder` tool to send the message.

If needed, use the `current_time` tool to compute a future time.

Remember:

you do not keep track of time.

You are part of a durable system that can wait hours, days, or weeks and still complete the task. The system will wake you when it is time to act. Your job is to determine what to do next.

The model autonomously chooses from these tools:

• schedule: Sleep until a given UTC time in ISO 8601 format

• reminder: Send a message as a reminder

• current_time: Return current UTC time in ISO 8601 format

The core interaction between GPT and the tools happens inside a Resonate function.

Durable Suspension and Durable Resumption

Besides the LLM, the highlight of this system is Resonate’s durable execution model.

When the assistant calls:

Resonate

• Suspends the computation

• Waits for hours, days, weeks, or months

• Resumes the computation－even on a different process

A procedural programming model on top of an event driven execution model. No databases, no queues, no timers, no polling—just functions and promises

Visit the repo README for installation and usage instructions.

Try it!

Production software that’s as tough as diamonds?

Deterministic Simulation Testing makes it possible.

Join us, Wednesday June 4th at 10AM PT as we explore the DST implementations of both the Resonate Server and the Resonate Python SDK.

RSVP to get a calendar invite.

RSVP

And join the Discord to stay informed.

Join Discord

Someone asked in the Resonate Discord:

What is the benefit of using Resonate over a job queue?

Persisting a job to a queue implies that there is no single place where the business logic of the application is defined. Essentially, a distributed application relies on various services or components to each play a part and then hand off the next step to another. In practice, this makes it very difficult for a single developer to learn, reason about, and contribute to the application’s business flow.

At a high level, Resonate offers a sequential-looking programming model built on top of a concurrent execution model (such as a job queue).

One could argue that a Resonate Function consisting of a single span or step, and a job on a traditional job queue, are functionally equivalent. The difference lies in the developer experience and the ease of reasoning about the application’s business logic.

It is also important to note that a Resonate Application Node can act as both a consumer and a producer between job queues — effectively serving as a processor. It naturally avoids head-of-line blocking issues and provides a durable bridge between queues.

Furthermore, Resonate can use job queues to pass messages between Resonate Servers and Application Nodes.