Lightbus vs Celery

Lightbus was conceived as a result of using Celery to communicate between multiple Python services.

Differences in principle

Celery is a task queue:

  • A task queue is tightly coupled. The dispatching code must know what needs to be done
  • A task queue typically doesn't return results

Lightbus is a bus:

  • A bus provides loose coupling. The dispatching code says what did happen, not what should happen (events)
  • A bus provides bi-directional communication (remote procedure calls)

Differences in practice

A number of pain points were identified with Celery that Lightbus aims to address. In particular:

  • Single vs multi-service – Celery is targeted as being a task queue for a service, rather than a means for multiple services to interact.
  • Conceptual overlap – The mapping between concepts in Celery and the underlying broker (AMQP at the time) is both unclear and overlapping. Lightbus provides a limited set of well defined concepts to avoid this confusion.
  • Non-sane defaults – Some Celery settings have non-sane defaults, making setup somewhat perilous at times. Lightbus provides sane defaults for most circumstances, and documentation specifically geared to certain use cases (metrics, event sourcing)
  • Tight coupling (as discussed above) – Celery tasks define the action to take, not what happened. Lightbus uses events, which describe happened, and listening services decide the action to take.
  • General feeling – Celery feels large and opaque, debugging issues was challenging. Lightbus aims to feel lightweight, with clear logging and debugging tools.