RabbitMQ topology

MailerQ leverages RabbitMQ queues and exchanges to handle email processing and message flows. This setup provides flexibility in how emails are managed and delivered and how results can be routed and processed. Below, we outline how to configure this topology, with configuration options tailored to your needs.

Core concepts in RabbitMQ

Queue: A list where messages wait until they are processed by a consumer. MailerQ uses an "outbox" queue from which it reads the emails that need to be sent.

Exchange: A distribution point in RabbitMQ where messages are published. The exchange determines, based on the routing key, which queue(s) will receive a message. MailerQ normally sends all its messages to the same exchange.

Routing Key: A specific key that is used when messages are published to RabbitMQ. MailerQ uses different routing keys for different types of messages, such as retries, and success and failure reports.

Binding: A link that directs messages from an exchange to a queue or even another exchange. Bindings can have a routing key pattern, so only messages with matching routing keys are delivered to the connected queue or exchange. Bindings between exchanges allow for advanced message flows by chaining exchanges with different routing logic.

Using these concepts you can setup advanced topologies that allow you to route delivery reports to your own processing tools, and send retries and received messages back into MailerQ.

The input queue

MailerQ reads its email instructions from a single, specified queue, commonly referred to as the "outbox" queue. All mails that need to be delivered should be placed in this queue. The following config file setting determines the name of this queue:

rabbitmq-queue: outbox

MailerQ only consumes from this queue, and does not directly post new messages to it. When MailerQ wants to send or reschedule mail, it sends it to an exchange with a routing key. With the right topology, such messages then end up in the outbox queue.

In older versions of MailerQ this setting was named rabbitmq-outbox, and was used to specify both the queue, and the routing key for publishing.

The output exchange

All results from email processing, including retries and failures, are sent back to RabbitMQ. Also inbound mails, bounces, and messages that are received via other ways (directory pickup, API, etc), are sent to the exchange. In the config file the exchange can be set with this setting:

rabbitmq-exchange: mailerq

By default, the name of the exchange is not configured, and messages are published to the "default exchange", which is a special exchange in RabbitMQ with limited routing capabilities: it treats the routing key as a queue name, and places each message in the queue with the name of the routing key. If such a queue does not exist, the message is lost. When you leave the exchange empty, all routing keys in the config file are effectively queue names.

For advanced routing options, it is strongly recommended to change this setting and use a named exchange. We also recommend to only let MailerQ publish messages to this exchange, and create additional exchanges and bindings for your own publish operations. This will simplify monitoring and debugging, as you can be certain that all traffic through the exchange is guaranteed to come from MailerQ.

Routing keys for rescheduling and retries

Routing keys are used for categorizing messages based on their purpose. MailerQ uses different routing keys for various operations, allowing messages to be routed to other tools or back into MailerQ. If you do not specify an exchange in the config file, the routing keys are effectively queue names.

The following config file settings can be used to set the routingkeys used for scheduling mails:

rabbitmq-outbox: outbox
rabbitmq-retries: retries

The "outbox" routing key is used to put a message in the outbox. The "retries" routing key has the same purpose, with the difference that it is only used to schedule retries, for example when a 4xx SMTP response code was received or when a mail server was otherwise unreachable.

The recommended topology is to bind both routing keys to the outbox queue: all rescheduled and retried messages do then indeed end up in the outbox again. Additionally, you can bind the "retries" routing key to a separate queue of your own, which allows you to monitor temporary failures.

Routing keys for delivery results

When a mail was delivered, or when it permanently failed, the result is published back to RabbitMQ with routing keys configurable by these settings:

rabbitmq-success:
rabbitmq-failure:

If you leave the settings empty, which is the default, MailerQ will not publish delivery results. Do not bind the routing keys to the outbox queue, because the delivery results are only meaningful for further processing by external tools, and not to be picked up again by MailerQ.

Temporary failures are not published back with one of these routing keys, but are sent back to the outbox with the special "retries" routing key. To monitor these retries, set up an extra binding for this routing key into one of your own queues.

Routing keys for inbound mails

MailerQ can handle inbound SMTP connections. All messages that are received on its inbound SMTP port are published to RabbitMQ, using routingkeys set by these config file options:

rabbitmq-inbox: outbox
rabbitmq-local:
rabbitmq-reports:
rabbitmq-refused:

The inbox routing key is for historical reasons standard set to "outbox", so inbound mail follows the same RabbitMQ routing as regular retried mail and is immediately scheduled for delivery. It is recommended to set this to a different value (like "inbox") and create an extra binding for this routing key to let the email flow back to the outbox. This has the same end result (inboud mail gets scheduled for delivery), but improves your monitoring capabilities.

The "local", "reports" and "refused" settings are optional. If you leave them empty, MailerQ simply publishes all incoming message with the inbox routing key. Once you do configure one or more of these extra routing keys, MailerQ starts analyzing and categorizing inbound messages, and publishes them with special routing keys based on this.

MailerQ internally keeps a list of "local" email addresses. This list can be edited via the management console. If you have configured a "local" routing key, and a mail comes in for an address on that list, the message is published with the "local" routing key.

If the local email also turns out to be not a normal type of email, but some kind of delivery report (or some other kind of report), the message is published with the "reports" routing key.

To summarize:

• inbox routing key: regular incoming messages
• local routing key: incoming messages for local recipient addresses
• reports routing key: incoming messages for local recipients that contain a delivery report
• refused routing key: incoming messages that were not accepted

The last listed routing key is "refused". It is used for messages that were not accepted. For example, if you configure MailerQ to listen to one or more SMTP ports, and you require incoming connections to be authenticated, you might receive messages over unauthenticated connections. These messages are rejected and do not end up in RabbitMQ. However, for debugging and/or security reasons you might still want to find out who is flooding your SMTP server. By assigning a value to "rabbitmq-refused", you instruct MailerQ to publish rejected messages with a special routing key so that you can route them to a queue where you can inspect them.

Delivery status notifications

In the above sections we described the queue from which MailerQ reads its outgoing messages (the outbox queue) and the exchange and routing key to which all incoming messages are published. However, besides handling incoming and outgoing messages, MailerQ can also construct and send email messages all by itself: Delivery Status Notifications (DSN).

A delivery status notification is an email message that is sent back to the original envelope address when a delivery fails (technically, it is also possible to send such notifications on successful delivery and/or when a message gets delayed, but in practice it is mostly used for failure notifications). By default MailerQ does not send such notifications because it uses result queues and JSON to report delivery results. However, if you add a "dsn" property to the JSON of an outgoing message, you can instruct MailerQ to send delivery status notifications too.

Technically, a delivery status notification is a regular email, and MailerQ simply posts a message to RabbitMQ when such an email has to be delivered. The routing key that is used can be controlled with this setting:

rabbitmq-dsn: outbox

The default setting is outbox, so it follows the same process as regular mails. If you want to intercept these notifications, or route them differently, you can install a different routing key.

Designing the right topology for MailerQ

MailerQ’s configuration file specifies queues and routing keys, and leaves it up to the user to set up this topology in RabbitMQ, with possible extra queues and bindings for handling results. This flexibility allows you to decide if and how to process messages routed by these keys, creating customized workflows that best fit your email processing needs. We encourage you to create your own topology, which gives you maximum control over the queues and bindings in RabbitMQ.

You have several options for setting up your topology:

• Using RabbitMQ’s management interface
• Leveraging RabbitMQ’s command-line utilities
• Creating a custom script or program
• Using our amqpinit tool
• Or by other means that suit your environment

Additionally, MailerQ can automatically create queues and bindings if desired.

Letting MailerQ set up the topology automatically

MailerQ can infer the desired topology from the config file and, upon startup, create exchanges, queues, and bindings based on this configuration. Note that this process is based on assumptions, as the config file defines routing keys but not specific queues or bindings. To enable this feature, use the following settings in your configuration file:

rabbitmq-declare:           true
rabbitmq-durable:           true
rabbitmq-lazy:              false

When rabbitmq-declare is set to true, MailerQ creates exchanges, queues and bindings at startup. You can control the persistence of these resources with rabbitmq-durable (to keep them after a RabbitMQ restart), and with rabbitmq-lazy you can opt to create lazy queues, which store messages on disk to ensure predictable performance under load.

Rules for automatic resource creation

When automatically setting up the topology, MailerQ follows these conventions:

• The exchange defined by rabbitmq-exchange is created as a direct exchange.
• If routing keys for rabbitmq-outbox, rabbitmq-failure, rabbitmq-success, rabbitmq-inbox, rabbitmq-local, rabbitmq-reports, rabbimq-refused, or rabbitmq-dsn are specified, queues are created with these names.
• The above queues are all bound to the exchange using their respective routing keys.
• For the rabbitmq-retries routing key no special queue is created, but only a binding from the exchange to the outbox queue.
• If the config variables rabbitmq-results or rabbitmq-retry exist, these queues are created and bound to the exchange with success+failure and retries routing keys, respectively.

The rabbitmq-results and rabbitmq-retry settings are retained for compatibility with earlier versions of MailerQ, where results and retries were directly queued rather than routed. These settings are no longer used when running MailerQ, but are still respected during initialization to create and bind queues.

Lazy queues explained

Setting rabbitmq-lazy to true makes MailerQ declare its queues as lazy. All messages are then written directly to disk, ensuring performance stability at the cost of some speed. With lazy queues, RabbitMQ is consistently slow, which is sometimes preferrable over being slow during peak loads only. For more details, see the RabbitMQ documentation on lazy queues.

Using AMQP-init for setting up the topology

If the standard topology that is created by MailerQ is not sufficient, you need to create the topology yourself. Our amqpinit tool can be of help. It reads JSON input holding the toplogy setup. The following input JSON can for example be used for setting up a straight forward topology:

[{
    "type": "exchange",
    "name": "mailerq",
    "exchangetype": "direct",
    "durable": true
}, {
    "type": "queue",
    "name": "outbox",
    "durable": true
}, {
    "type": "queue",
    "name": "failures",
    "durable": true,
    "arguments": {
        "x-max-length": 10000
    }
}, {
    "type": "queue",
    "name": "success",
    "durable": true,
    "arguments": {
        "max-max-length": 10000
    }
}, {
    "type": "binding",
    "exchange": "mailerq",
    "queue": "outbox",
    "routingkey": "outbox",
}, {
    "type": "binding",
    "exchange": "mailerq",
    "queue": "outbox",
    "routingkey": "retries"
}, {
    "type": "binding",
    "exchange": "mailerq",
    "queue": "outbox",
    "routingkey": "inbox"
}, {
    "type": "binding",
    "exchange": "mailerq",
    "queue": "failures",
    "routingkey": "failure"
}, {
    "type": "binding",
    "exchange": "mailerq",
    "queue": "success",
    "routingkey": "success"
}

The above JSON takes care of the following:

• It begins with the basics: the input queue ("outbox"), and the output exchange ("mailerq").
• It has additional queues for successes and failures, with extra "max-queue-length" property to prevent RabbitMQ overflow if the process to read the queues gets stuck.
• There are bindings for "outbox" and "retries" so that rescheduled or retried messages are routed back to the outbox.
• There is a binding that puts inbound mail in the outbox (for a setup where SMTP is used for submission).
• There are bindings so that success and failure reports are routed to their queues.

If you use AMQP-init, or a different technology, for setting up the topology, remember to change your MailerQ config and set rabbitmq-declare to false, because you no longer need MailerQ to do this for you.