Python job queue · Redis · asyncio
Jobs that survive the worker running them.
Every state change is one atomic Lua script. Every running job holds its own lock. When a worker dies mid-job, a stalled sweep hands the work to another worker. Delivery is at-least-once.
from toro import Queue, Worker
queue = Queue("emails", url="redis://localhost:6379")
await queue.add(
"welcome",
{"to": "ada@example.com"},
attempts=3,
)
async def send(job):
return await smtp.send(job.data["to"])
await Worker("emails", send, concurrency=20).run()
States
Eight states, one atomic move between them
A job is in exactly one of them at a time, and every move is a single Lua script, so a crash cannot leave a job in two of them or in none. The tones are the dashboard's own.
- wait
- Ready to run, waiting for a free worker.
- delayed
- Scheduled for later, promoted when it comes due.
- held
- Behind another job on the same concurrency key, holding no slot.
- active
- Claimed by a worker, running under a lock that worker renews.
- waiting-children
- A flow parent, parked until its last child settles.
- completed
- Finished. returnvalue holds what the processor returned.
- failed
- Out of attempts. failed_reason holds the error.
- cancelled
- Stopped on purpose. Terminal, counted apart from failures.
The normal path is wait → active → completed. A failure with attempts left goes back to wait, or to delayed when a backoff applies, and only the last one lands in failed.
Reliability
What happens when a worker dies
A claimed job is held under a lock that its worker renews while the processor runs. A process that disappears stops renewing, and the next sweep puts its jobs back.
- The lock proves the worker is alive
- A renewer refreshes it every lock_renew_time, so a lock expires when renewal stops: the process died, or its event loop was blocked past lock_duration.
- Recovery is a sweep, not a timeout
- Every worker looks for jobs whose lock expired and re-enqueues them at their stored priority, throttled so the fleet sweeps about once per interval.
- The result commits once
- A late finish from the dead process is rejected by its token: the handler may run twice, the job settles once.
await Worker(
"emails", send,
lock_duration=30_000, # the lease a running job holds
lock_renew_time=15_000, # renewed this often
stalled_interval=30_000, # swept for this often
max_stalled_count=1, # recoveries before it fails
).run()
API
Common tasks, and the call for each
Retries, scheduling, rate limits, transactional enqueue, cancellation and flows, configured where the job is enqueued or the worker is started.
await queue.add(
"report", data,
attempts=5,
backoff={"type": "exponential", "delay": 1000},
)
Features
Everything a queue is expected to have
None of this is an add-on or a plugin. It is in the library and documented, and every state change it makes runs inside the same atomic scripts that move jobs between states.
Retries with backoff
Fixed or exponential, per job, bounded by the attempt count. A failed attempt goes back with its logs and attempt count kept.
Priorities and delays
Enqueue ahead of the queue, or for later. Delayed jobs promote themselves when due.
Cron and repeatable
Every occurrence schedules the next, so a missed tick cannot compound into a storm.
Deduplication
A custom id makes add() idempotent while that job is kept: a second one is ignored rather than queued twice.
Rate limits
A cap across every worker on the queue, not per process, spent only once a job is actually claimed.
Concurrency keys
Serialise jobs that touch the same thing while everything else keeps flowing.
Global concurrency
One ceiling on active jobs across every worker, released even when a worker crashes.
Flows
Parent and child trees with fan-out, fan-in, per-child failure policy and flow-aware retry.
Metrics
Per-minute counters and lifetime totals in Redis, exported as OpenMetrics for Prometheus.
Sync or async processors
A plain def runs in a thread pool the worker owns, sized to its concurrency, so the loop stays free to renew locks.
Transactional enqueue
Collect jobs while a database transaction is open and send them in one pipelined write once it commits.
Cancellation
cancel_job stops a running async job at its next await, so finally blocks run and it lands in cancelled rather than failed.
Flows
Fan out, fan in, without the plumbing
A flow is a parent job whose children run first. The parent waits in waiting-children until they finish, then runs with their results.
- The tree is the unit
- Retry a parent and the failed branch re-runs, not the work that already succeeded.
- Failure is a policy
- on_fail decides per child: fail the parent, or carry on without it.
- Children stay out of the way
- Listings show root jobs, so a flow reads as one thing and opens into its tree.
parent = await queue.add_flow(
"send-digest", {"user": 42},
children=[
FlowChild("render-header", {}),
FlowChild("render-body", {}),
FlowChild("render-footer", {}, on_fail="continue"),
],
)
async def send_digest(job):
# {child_id: returnvalue}
parts = await job.children_results()
return smtp.send("".join(parts.values()))
Dashboard
Mounted, not run
matador is a FastAPI app. It goes inside the application you already deploy, behind the auth you already have, so there is no second service to run and no second login to hand out.
from fastapi import FastAPI, Depends
from matador import create_app
app = FastAPI()
app.mount(
"/admin/queues",
create_app(
["emails", "billing"],
connection=redis, # the host client: a mounted app never runs its own shutdown
dependencies=[Depends(admin)],
can_mutate=lambda r: getattr(r.state, "role", None) == "admin",
),
)
- Read-only is a predicate
- can_mutate decides per request, and when it says no the controls are never drawn, not merely refused.
- Scrapeable
- /metrics speaks OpenMetrics, so the queue reaches Prometheus without an exporter.
- Live without a socket
- Server-sent events from one shared Redis subscription, released when the last viewer leaves.
Contract
What you are depending on
- Version
- 1.1.1, frozen API
- Python
- 3.10 to 3.13
- Redis
- 6.2 and later
- Tests
- 1200+, most on real Redis
- Dependencies
- redis, croniter
- Licence
- MIT