Compression and configurable retry intervals

The upcoming version of MailerQ - the expected version number is 4.2.4 - will have two small (but nice) new features. The communication with RabbitMQ can be gzip compressed, which brings down the amount of data that is sent over the internal network between MailerQ and RabbitMQ, and the retry intervals are no longer hardcoded, but can be configured in the config file, or on a per-message basis in the JSON.

Compression

With the upcoming version, MailerQ can handle compressed data from RabbitMQ. To recognize whether a message is compressed or not, MailerQ inspects the envelope that is wrapped around each message consumed from RabbitMQ. If the envelope contains a "content-encoding" field, and when it is set to "gzip", MailerQ expects the message payload to be gzip compressed, and will decompress it first before it treats the message as a JSON object.

In previous versions of MailerQ, we completely ignored the AMQP envelope fields, and always expected the payload to be JSON encoded. With this new version of MailerQ, we now do check the envelope, and treat the message as a gzip compressed message if the "content-encoding" field is set to "gzip". If you already used the "content-encoding" field from the AMQP envelope to store your own message meta data, you should change this because it might conflict with how MailerQ handles it.

If you inject messages by publishing them to RabbitMQ directly, you can make use of this new feature by compressing the JSON data first. You can then publish the compressed data to RabbitMQ, as long as you make sure that you set the "content-encoding" field in the AMQP envelope to "gzip".

If MailerQ needs to publish data to RabbitMQ (for example because an incoming message has to be sent to the "inbox" queue, or a delivery result has to be published to one of the result queues), it can use compression too. In the config file you can set the "rabbitmq-encoding" field to "gzip". If you do this, MailerQ will compress all data that it publishes to RabbitMQ.

If you have your own scripts that read out queues that are written by MailerQ, and you have compression enabled in the MailerQ config file, you should be able to handle compressed data. We advise to modify your scripts and check the AMQP envelope for the "content-encoding" field. If it is set to "gzip", you should decompress the message first before you parse it as json.

Configurable retry intervals

The next nice new upcoming feature are the configurable retry intervals. Up to now, MailerQ had its own hardcoded retry intervals. When a mail could not immediately be delivered (for example because of greylisting), MailerQ started a second attempt after 10 minutes. This interval was longer for later attempts: the third attempt used an one hour interval, while from the sixth attempt onwards, new attempts were started with a three hour interval.

This interval is now configurable. In the config file you can set a variable holding the intervals in seconds. This "retry-interval" variable should be set to a comma separated list of interval lengths. If you do not explicitly set it, a default value of "600,600,1800,1800,3600,3600,7200" is used. The last value from the list holds the interval for all subsequent attempts.

It is possible to override the interval in the per-message JSON. You can add a "retries" property to the json, and set it to a JSON array holding the number of seconds.

This interval settings allows you to use different retry intervals for transactional mails (messages that you want to retry fast because they are important to be delivered as soon as possible), and for commercial mass mailings (which are not that important).