Result message queues

MailerQ picks up messages from the outbox queue, tries to send them out and then publishes the results back to result queues. Other programs or scripts can read out these result queues and further process the failures and succesful deliveries. The messages on the result queues are all JSON encoded.

Different result queues

MailerQ uses four different queues to which results are published. Based on the type of result (success, failure or retry) a message is published to one or more of these queues:

• The results queue
• The success queue
• The failures queue
• The retry queue

The "results" queue is used by MailerQ to publish all the results: both the successful results as well as the failures. The "success" and "failures" queues are used for just the successes and just the failures. These queues receive a subset of the messages published to the "results" queue. The "retry" queue is used for transient results for deliveries that have not yet failed or succeeded, and that are going to be retried.

Depending on your needs, you can enable or disable these different queues (for example, if the only thing you're interested in are the failures, you do not configure the results, success and retry queues, and only set the failures queue). The result queues can be configured in the RabbitmQ section in the config file.

Example scenario

A typical delivery scenario is this: you first publish your JSON message to the outbox queue. MailerQ picks up this message and tries to send it. If this send attempt fails (because the receiving server rejected it with a soft error), MailerQ modifies the initial JSON to include info about this first soft error, and publishes it back to the outbox queue to retry the delivery later. The message is also sent to the retry queue to notify external programs that the message will be retried.

After a while, MailerQ picks up the message from the outbox queue again for a second attempt. This time the delivery succeeds and the message is modified once more to also contain info about this correct delivery, and the message is published to both the success queue and the generic results queue.

Understanding the results

The messages that MailerQ publishes to the results queues are JSON encoded, and hold the same properties as the original message that was read from the outbox queue, including all the custom properties that you added (thus: if you added some custom identifier to the initial JSON object, it will also be present in the JSON object that ends up in the result queue.

The "mime" property is often stripped from the initial JSON before it is published to a result queue, so that these queues do not take up too much space from RabbitMQ. However, if you want to keep the mime data in the JSON, you can add a "keepmime" property. For more info about the supported properties, check the JSON specification for outgoing messages.

Besides stripping the MIME data, MailerQ also adds a "results" property to the JSON. This "results" property holds an array with information about each delivery attempt that was taken. A result message will thus look something like this:

{
    "envelope": "my-sender-address@my-domain.com",
    "recipient": "info@example.org",
    "results": [
        {
            "state": "intro",
            "result": "error",
            "time": "2016-02-04 13:45:15",
            "mta": "mta1.example.com",
            "from": "1.2.3.4",
            "to": "10.11.12.13",
            "messages": 1,
            "sent" : {
                "connection": 355,
                "data": 0
            },
            "code": 451,
            "status": "5.0.0",
            "description": "Please come back later",
        },
        { 
            "state": "message",
            "result": "accepted",
            "time": "2016-02-04 14:01:25",
            "mta": "mta2.example.com",
            "from": "1.2.3.4",
            "to": "10.11.12.14",
            "messages": 4,
            "sent" : {
                "connection": 5876,
                "data": 4923
            },
            "code": 250,
            "status": "2.0.0",
            "dsn": true,
            "description": "Requested mail action okay, completed",
        }
    ]
}

The "results" property holds an array of JSON objects. Each object contains the result of one delivery attempt. Every result object can have the following properties:

Note that some properties are optional. For example, the properties "to", "from", "mta" and "messages" are only used if an actual TCP connection was set up and are not present if the mail failed because of a DNS lookup failure. The "code", "status" and "description" properties are only used when the remote server returned an error code and/or human readable description to explain why the message was or was not accepted.

The "mta" property holds the name of the MTA that accepted the message. To get this MTA name, MailerQ reads the answer from the initial welcome message that the receiving server sent right after the connection was set up.

The "dsn" property is only used for succesful deliveries and tells you whether the incoming server supports the DSN extension. If this is set to true, and you had also included a DSN setting in your input JSON to request a success notification, you can expect a notification from the server that accepted the message.

Added properties

For logging purposes, MailerQ also extracts some data from the MIME when publishing the emails back. This way, these properties can be logged using the flexible log format.

{
    "envelope": "my-sender-address@my-domain.com",
    "recipient": "info@example.org",
    "results": [
        { 
            "state": "message",
            "result": "accepted",
            "time": "2016-02-04 14:01:25",
            "mta": "mta2.example.com",
            "from": "1.2.3.4",
            "to": "10.11.12.14",
            "messages": 4,
            "code": 250,
            "status": "2.0.0",
            "dsn": true,
            "description": "Requested mail action okay, completed",
        }
    ],
    "extracted": {
        "from": "my-sender-address@my-domain.com",
        "to": ["info@example.org", "other@example.org"],
        "subject": "Test Subject",
        "headers": {
            "x-example-property": "some-mime-property"
        },
        "envelope": { 
            "domain": "bounce.my-domain.com", 
            "local": "my-sender-address" 
        },
        "recipient": { 
            "domain": "example.org", 
            "local": "info" 
        } 
    }
}

As is shown in the example, some properties are exposed in a parsed way (from, to, subject), as well as unparsed exposure of all the MIME-headers that were present in the (personalized) email.

Delivery states

MailerQ logs the state in which the error occurs (this is the "state" property), and the type of result in that state. An email that is processed by MailerQ goes through the following states:

During all of the above states errors might occur. If this happens, the state of the delivery and the type of error is set added to the result JSON. A successful delivery has gone through all the states, and is published to the result queue with the property "state" set to "message" and "result" set to "accepted".

Result types

During all the above states errors can occur. The type of error is logged in the "result" property. It always has one of the following values:

The SMTP protocol requires communication between two mail servers: a sending server and a receiving server. MailerQ is the sending server, and sends instructions to the receiving server (like "MAIL FROM", "RCPT TO", et cetera). After each of these instructions it waits for an answer from the remote server.

If the answer is not received in time, the "timeout" result is used. It is also possible that the receiver did sent back an answer, but that the answer could not be recognized as a valid SMTP response message. In such a case we use the "invalid" result type. When the TCP connection was lost while waiting for an answer the "lost" type is result, and the "error" type is used in case the receiver server sent back a valid SMTP answer, but the answer was that the message was rejected.

If MailerQ's instruction was accepted, the delivery moves to the next state, and no result is logged in the JSON object.

Special combinations

Not every result state is linked to a state in the SMTP protocol. The following combinations can also occur, and should be interpreted in a special way:

MailerQ does its best to fill in an appropriate error code in the result object too. Be aware that the error code and status code sometimes come from the remote server and is blindly copied by MailerQ to the result JSON, while in other situations it is generated by MailerQ.

Secure connections

If the receiving server supported STARTTL, and MailerQ managed to secure the connection, the result object also holds an "cipher" and "certificate" key. The "cipher" holds the settings of the encryption algorithm that was used, while the "certificate" holds the settings of the SSL certificate that was issued by the receiving server. Both the "cipher" as the "certificate" are stored as JSON objects, holding all sorts of sub values.

{
    "envelope": "my-sender-address@my-domain.com",
    "recipient": "john.doe@gmail.com",
    "results": [
        { 
            "state": "message",
            "result": "accepted",
            "time": "2016-07-20 14:01:25",
            "mta": "mx.gmail.com",
            "from": "1.2.3.4",
            "to": "10.11.12.14",
            "messages": 1,
            "code": 250,
            "status": "2.0.0",
            "dsn": false,
            "description": "Requested mail action okay, completed",
            "cipher": {  
                "name": "ECDHE-RSA-AES128-GCM-SHA256",
                "bits": 128,
                "algbits": 128,
                "version": "TLSv1\/SSLv3",
                "id": 49199
            },
            "certificate": {  
                "notbefore": "2016-07-13 14:40:53",
                "notafter": "2016-10-05 14:17:00",
                "version": 3,
                "verified": true,
                "result": "ok",
                "keyusage": 1,
                "self-signed": false,
                "subject": {  
                    "common": "mx.google.com",
                    "country": "US",
                    "locality": "Mountain View",
                    "state": "California",
                    "organization": "Google Inc"
                },
                "issuer": {  
                    "common": "Google Internet Authority G2",
                    "country": "US",
                    "organization": "Google Inc"
                },
                "altnames": [  
                    "gmr-smtp-in.l.google.com",
                    "gmr-mx.google.com",
                    "gmail-smtp-in.l.google.com",
                    "aspmx5.googlemail.com",
                    "aspmx4.googlemail.com",
                    "aspmx3.googlemail.com",
                    "aspmx2.googlemail.com",
                    "aspmx.l.google.com",
                    "alt4.gmr-smtp-in.l.google.com",
                    "alt4.gmail-smtp-in.l.google.com",
                    "alt4.aspmx.l.google.com",
                    "alt3.gmr-smtp-in.l.google.com",
                    "alt3.gmail-smtp-in.l.google.com",
                    "alt3.aspmx.l.google.com",
                    "alt2.gmr-smtp-in.l.google.com",
                    "alt2.gmail-smtp-in.l.google.com",
                    "alt2.aspmx.l.google.com",
                    "alt1.gmr-smtp-in.l.google.com",
                    "alt1.gmail-smtp-in.l.google.com",
                    "alt1.aspmx.l.google.com",
                    "mx.google.com"
                ]
            },
        }
    ]
}

Overriding errors

With the Response Pattern system, properties can be added to the result json. This is useful for your own classification and processing of messages, but there are also some special properties that can be used to override error handling logic.

These properties are not set by MailerQ itself, but can be set by setting these properties in a Response Pattern. If they are not present, MailerQ tries to figure out these properties by reasoning about the response it got from the server.