Resonate is also a Task Framework

Recursive function executions to distribute a calculation? No way!?

Did you know that Resonate is also an easy to use generic task framework? Resonate comes with everything needed to distribute tasks to workers and await results out-of-the-box.

Clone the resonator repo to follow along, you will need to have the following prerequisites installed:

• python (and optionally uv)

• resonate

Here is a preview of what we are building.

Diving In

Imagine you need to do a calculation, but not just any calculation. Imagine you need to do a calculation so large that it would take a single computer way too long to compute. But lucky for you you have a set of worker computers and you know a few things about your calculation.

1. All operands are integers, and

2. All operations are binary operations that are closed under the set of integers

With these rules we can break our large calculation up into many smaller calculations and distribute them. Let’s build a distributed calculator, let’s build Resonator!

First, let’s define our operations.

These operations need to be registered with Resonate, this will make them available on the workers. Once registered, tasks can be called with run.

Tasks can run in parallel.

And tasks can be chained with other tasks through simple argument passing.

Additionally, tasks can be routed to named queues with send_to.

And tasks can be routed to a preferred process, if available, falling back to any worker consuming from the named queue.

The poll scheme indicates to Resonate to send tasks via the long poll transport which comes built-in to the Resonate server. We plan to support more transports (including queues such as NATs and Amazon SQS) in the future.

All Together Now

Let’s put all of these building blocks together to create Resonator!

First, we are going to need a parser that can interpret a mathematical expression and split it into nested subexpressions. This is left as an exercise for the reader (admittedly I had ChatGPT write this for me). Let’s just assume we have a parse function that returns an expression represented as a tuple containing an operator, a left-hand operand, and a right-hand operand.

Next, let’s write a recursive function leveraging everything we have so far.

Here we match the expression, if the expression has already been evaluated to an integer there is nothing left to do but return it. If the expression contains an operator and two operands we do the following:

1. Yield the lhs and rhs operands to the exp task queue with preference. If there is a worker consuming from the queue with the ids “lhs” and “rhs” respectively that worker will process the task, otherwise the task will be processed by any available worker.

2. Await the result of the lhs and rhs tasks. Note that by awaiting following both invocations we are performing the evaluations concurrently.

3. Yield the operation to the ops task queue. This task will be processed by any available worker. Here, we are executing the function by it’s registered name which corresponds to the op symbol (+, -, *).

With Resonate, there is no distinction between a workflow and a step, everything is simply a function. We can call clc from our local computer and distribute tasks to workers consuming from the exp queue, which recursively call the exact same clc function! Once all operands are fully evaluated tasks can be sent to the ops task queue where the corresponding operation function is called.

If you are interested in learning more about Resonate or talking about distributed systems in general join our discord, check out our website, and subscribe: