Shrinking Infrastructure: Task Queues on PostgreSQL by Jan Bjørge | PyConSe | 2025

Most Python teams already run PostgreSQL—but when it comes to background jobs, they often add Redis, RabbitMQ, or Celery on top. pgqueuer takes a different approach: it turns Postgres itself into a complete task queue. At its core, pgqueuer uses simple tables to capture jobs and their metadata. Workers safely lease jobs with FOR UPDATE SKIP LOCKED, a Postgres feature that enables concurrent consumers without collisions. Idle workers wake instantly on new jobs via LISTEN/NOTIFY, while a fallback polling loop ensures progress even if notifications are lost. On top of this foundation, pgqueuer adds the pieces needed in production: exactly-once dequeueing, heartbeats and cancellation for long-running tasks, and retry semantics for failure recovery. A built-in CLI dashboard provides live insight into throughput, latency, and error counts—no extra services required. This talk walks through pgqueuer’s design and shows how PostgreSQL features can form a reliable, observable job system. Attendees will leave with a understanding of how to run production-ready background jobs directly on the database they already trust. #pyconse #pyconse2025