Handling delayed messages

MailerQ’s messaging system includes support for delayed message handling — a feature critical for scenarios such as managing graylisting requirements, enforcing send limits, or staging messages for later delivery. RabbitMQ, however, is fundamentally a FIFO (First-In, First-Out) system, not designed with inherent delay capabilities. Messages sent to a RabbitMQ queue are processed as soon as they are available to consumers, making it challenging to hold messages back for a specific period.

To work around these limitations, MailerQ employs a solution by leveraging RabbitMQ’s dead-letter exchanges and message timeouts to create the effect of delayed message processing. This article covers the configuration, functionality, and limitations of this delay mechanism, along with an alternative solution using the amqpdelayer application for more efficient delay management.

How MailerQ delays messages with RabbitMQ

MailerQ implements delays by routing messages to a series of “delay queues,” each configured with a specific timeout. Here's how this process works in steps:

Parking messages in dead-end queues: When MailerQ determines that a message requires a delay, it places the message in a “dead-end” queue — a queue that is not consumed by any application and thus theoretically holds the message indefinitely. However, this queue has a timeout setting, so once the specified time elapses, RabbitMQ automatically expires the message.

Dead-letter routing: Each delay queue is also set up with a dead-letter exchange and routing key that directs expired messages back to MailerQ. When the message timeout is reached, RabbitMQ automatically forwards the expired message to MailerQ via the dead-letter exchange. MailerQ then decides if the message delay period has elapsed. If further delay is required, MailerQ moves the message to a new delay queue with a shorter timeout. This process continues across several delay queues until the delay period has ended. Finally, when no more delay is needed, the message is routed back to the outbox for processing.

Powers-of-two queue system: MailerQ sets up delay queues based on powers of two (1 second, 2 seconds, 4 seconds, etc.) to enable flexible delay durations. For example, if a message needs a 10-minute (600-second) delay, MailerQ places it first in a queue with a 512-second timeout (the largest power of two under 600), followed by a 64-second queue, and so on until it reaches the outbox with no delay.

Configuring delayed queues in MailerQ

The delay queues and bindings are automatically created when MailerQ starts. If required, settings in the MailerQ configuration can be modified to control the behavior and structure of these queues:

• delayed-queues: Sets the number of delay queues, typically 11 by default, covering timeouts up to 1024 seconds (powers of two from 1 to 1024 seconds).
• delayed-exchange: Defines the exchange used for routing delayed messages (defaults to the value of the rabbitmq-exchange setting).
• delayed-declare: Controls whether MailerQ should create delay queues on startup (default is true).
• delayed-durable and delayed-lazy: Determine whether the created queues are durable or lazy.
• delayed-type: Specifies the queue type, which can be either classic or quorum.

These configurations allow MailerQ to manage and customize delay queues, supporting flexible delay times while maintaining control over queue durability and memory usage.

Pros and cons of using delay queues

Using delay queues within RabbitMQ has several benefits and drawbacks:

Integrated and simple setup: The delayed queue system is part of RabbitMQ and MailerQ’s configuration. When MailerQ starts, it automatically creates the necessary delay queues, and messages are managed without additional infrastructure.

Reliable delays: By setting exact timeouts in seconds, MailerQ can ensure that delayed messages appear in the outbox queue at the desired time, meeting application-specific scheduling needs.

Resource-intensive: Every delay cycle involves messages expiring, being forwarded to the next queue, and eventually reappearing in the outbox. This increases the workload on RabbitMQ, MailerQ, and the network, particularly with frequent or high-volume delays.

Not designed for long-term storage: RabbitMQ is optimized for real-time message flows, not for long-term message storage. Using RabbitMQ to hold messages for extended periods can affect performance.

Queue complexity: The additional delay queues make the queue topology more complex, especially with multiple queues for various timeouts. This may complicate queue management and monitoring.

Alternative solution: amqpdelayer for efficient delayed messaging

For high-load or longer delays, an alternative approach is using amqpdelayer, a custom application designed to handle delayed messages outside of RabbitMQ. It reads delayed messages from RabbitMQ and holds them temporarily within its own storage until the delay period is complete. After the specified delay, it sends the message back to RabbitMQ for final processing, significantly reducing the workload on RabbitMQ by minimizing repeated message forwarding and queue cycling.

Since amqpdelayer stores messages independently, RabbitMQ’s delay queues don’t experience the constant expiry and movement of messages, resulting in a lighter load on RabbitMQ and MailerQ and improved overall efficiency.

Topology adjustments when using amqpdelayer

Integrating amqpdelayer into your mail stack requires adjustments to the RabbitMQ topology. The goal is to route all retries and reschedules to amqpdelayer, and after their delay back to the outbox. This can be achieved in many ways, but our recommendation is to setup an extra exchange and queue, while keeping the same routing keys. This gives the best monitoring and debugging options.

Separate exchanges for applications: Give each application its designated exchange. MailerQ writes to the "mailerq" exchange, while amqpdelayer writes to a separate "amqpdelayer" exchange.

Separate queues for applications: Each application gets its own input queue. MailerQ reads its input from the "outbox" queue, and amqpdelayer from the "delays" queue.

Routing delayed messages to amqpdelayer: The normal bindings between the "mailerq" exchange and the "outbox" queue should be removed, and replaced with two similar bindings between the "mailerq" exchange and the "delayed" queue. Add a binding for the routing key "outbox" and one for routing key "retries".

Routing processed messages to mailerq: Add two new bindings from the "amqpdelayer" exchange to the "outbox" queue. Use the routing key "outbox" and "retries" here too.

The above changes ensure that messages are still being moved around with the same routing keys, but are first processed by amqpdelayer before they are returned to MailerQ.

Adjustments to the MailerQ config file

Disable queue declaration: Because of the custom topology, you can no longer rely on MailerQ setting up the topology. The variables rabbitmq-declare and delayed-declare should therefore both be set to false.

Reduce number of delayed queues: MailerQ no longer has to pump messages around to the delayed queues, so the setting delayed-queues can be set to 0.

Adjustments to the AMQPDelayer config file

The config file of amqpdelayer should be modified to make it read from the "delayed" queue, and publish to the "amqpdelayer" exchange, while preserving routing keys. You especially need to pay attention to these settings:

• The variable rabbitmq-exchange in the amqpdelayer config file should be set to amqpdelayer.

• The variable rabbitmq-routingkey should be empty. This will make amqpdelayer publish the output messages with their original routing key. This can be helpful for debugging.