toro 1.1.1

Python job queue · Redis · asyncio

Jobs that survive the worker running them.

Every state change is one atomic Lua script. Every running job holds its own lock. When a worker dies mid-job, a stalled sweep hands the work to another worker. Delivery is at-least-once.

pip install toro-queue matador-dashboard MIT · 1.1.1
Quick start
from toro import Queue, Worker

queue = Queue("emails", url="redis://localhost:6379")

await queue.add(
    "welcome",
    {"to": "ada@example.com"},
    attempts=3,
)

async def send(job):
    return await smtp.send(job.data["to"])

await Worker("emails", send, concurrency=20).run()

States

Eight states, one atomic move between them

A job is in exactly one of them at a time, and every move is a single Lua script, so a crash cannot leave a job in two of them or in none. The tones are the dashboard's own.

wait
Ready to run, waiting for a free worker.
delayed
Scheduled for later, promoted when it comes due.
held
Behind another job on the same concurrency key, holding no slot.
active
Claimed by a worker, running under a lock that worker renews.
waiting-children
A flow parent, parked until its last child settles.
completed
Finished. returnvalue holds what the processor returned.
failed
Out of attempts. failed_reason holds the error.
cancelled
Stopped on purpose. Terminal, counted apart from failures.

The normal path is wait → active → completed. A failure with attempts left goes back to wait, or to delayed when a backoff applies, and only the last one lands in failed.

Reliability

What happens when a worker dies

A claimed job is held under a lock that its worker renews while the processor runs. A process that disappears stops renewing, and the next sweep puts its jobs back.

The lock proves the worker is alive
A renewer refreshes it every lock_renew_time, so a lock expires when renewal stops: the process died, or its event loop was blocked past lock_duration.
Recovery is a sweep, not a timeout
Every worker looks for jobs whose lock expired and re-enqueues them at their stored priority, throttled so the fleet sweeps about once per interval.
The result commits once
A late finish from the dead process is rejected by its token: the handler may run twice, the job settles once.
Worker options
await Worker(
    "emails", send,
    lock_duration=30_000,     # the lease a running job holds
    lock_renew_time=15_000,   # renewed this often
    stalled_interval=30_000,  # swept for this often
    max_stalled_count=1,      # recoveries before it fails
).run()

API

Common tasks, and the call for each

Retries, scheduling, rate limits, transactional enqueue, cancellation and flows, configured where the job is enqueued or the worker is started.

Retries with backoff
await queue.add(
    "report", data,
    attempts=5,
    backoff={"type": "exponential", "delay": 1000},
)

Features

Everything a queue is expected to have

None of this is an add-on or a plugin. It is in the library and documented, and every state change it makes runs inside the same atomic scripts that move jobs between states.

Retries with backoff

Fixed or exponential, per job, bounded by the attempt count. A failed attempt goes back with its logs and attempt count kept.

Priorities and delays

Enqueue ahead of the queue, or for later. Delayed jobs promote themselves when due.

Cron and repeatable

Every occurrence schedules the next, so a missed tick cannot compound into a storm.

Deduplication

A custom id makes add() idempotent while that job is kept: a second one is ignored rather than queued twice.

Rate limits

A cap across every worker on the queue, not per process, spent only once a job is actually claimed.

Concurrency keys

Serialise jobs that touch the same thing while everything else keeps flowing.

Global concurrency

One ceiling on active jobs across every worker, released even when a worker crashes.

Flows

Parent and child trees with fan-out, fan-in, per-child failure policy and flow-aware retry.

Metrics

Per-minute counters and lifetime totals in Redis, exported as OpenMetrics for Prometheus.

Sync or async processors

A plain def runs in a thread pool the worker owns, sized to its concurrency, so the loop stays free to renew locks.

Transactional enqueue

Collect jobs while a database transaction is open and send them in one pipelined write once it commits.

Cancellation

cancel_job stops a running async job at its next await, so finally blocks run and it lands in cancelled rather than failed.

Flows

Fan out, fan in, without the plumbing

A flow is a parent job whose children run first. The parent waits in waiting-children until they finish, then runs with their results.

The tree is the unit
Retry a parent and the failed branch re-runs, not the work that already succeeded.
Failure is a policy
on_fail decides per child: fail the parent, or carry on without it.
Children stay out of the way
Listings show root jobs, so a flow reads as one thing and opens into its tree.
Parent and children
parent = await queue.add_flow(
    "send-digest", {"user": 42},
    children=[
        FlowChild("render-header", {}),
        FlowChild("render-body", {}),
        FlowChild("render-footer", {}, on_fail="continue"),
    ],
)

async def send_digest(job):
    # {child_id: returnvalue}
    parts = await job.children_results()
    return smtp.send("".join(parts.values()))

Dashboard

Mounted, not run

matador is a FastAPI app. It goes inside the application you already deploy, behind the auth you already have, so there is no second service to run and no second login to hand out.

Mounting
from fastapi import FastAPI, Depends
from matador import create_app

app = FastAPI()
app.mount(
    "/admin/queues",
    create_app(
        ["emails", "billing"],
        connection=redis,  # the host client: a mounted app never runs its own shutdown
        dependencies=[Depends(admin)],
        can_mutate=lambda r: getattr(r.state, "role", None) == "admin",
    ),
)
Read-only is a predicate
can_mutate decides per request, and when it says no the controls are never drawn, not merely refused.
Scrapeable
/metrics speaks OpenMetrics, so the queue reaches Prometheus without an exporter.
Live without a socket
Server-sent events from one shared Redis subscription, released when the last viewer leaves.
/admin/queues The matador dashboard showing a queue's jobs, states and throughput matador, reading a toro queue Source →

Contract

What you are depending on

Version
1.1.1, frozen API
Python
3.10 to 3.13
Redis
6.2 and later
Tests
1200+, most on real Redis
Dependencies
redis, croniter
Licence
MIT
Read the source pip install toro-queue matador-dashboard