Faster troubleshooting with JSON messages in results queues
Log lines contain useful information about your message delivery, delivery errors and overall performance. Tracking and monitoring the information provided by log lines helps to identify and rectify any delivery problems, which is why they are a valuable asset in protecting your deliverability. Effective logging, however, can become quite challenging if you do not have the right logging standards in place and if you are restricted by your application output. MailerQ writes basic log lines to log files, but also logs JSON result objects to a results queue. This logging configuration has several added benefits for large senders, which we will discuss in this blog post.
Downsides of basic log lines
Basic 'unstructured' log lines are automatically produced texts that show what happens in the background of an application. It keeps track of all the 'hidden' events. Below you can find an example of a basic log line.
Log line:
[2018-08-29 07:20:16] message:accepted 1 [100.20.20.20] [100.30.30.30] [ "a", "b" ] 250 2.0.0 Ok There are a few problems with this basic log line configuration:
It is hard to read and to understand for other applications and systems.
One of the largest downsides of the basic log lines, is that they are hard to read and to understand on their own by other applications and systems. In order to make sense of it, in many cases, additional tools are used for parsing. Since many of these tools interpret these log files in their own way this might propose an issue, because it could mean that the information the tools provide is a somewhat clouded interpretation of the actual information of the log lines.
It comes in different formats
Basic log lines come in various different formats, which adds to why they can be so hard to read and understand for other applications. Depending on the server, log lines have their own prefix styles, object and process names, field orders and representations. There is no universal way to do this which can easily cause issues. It's like two people talking to each other in a different language that neither of them can understand.
Difficult to filter through the results
Without parsing, the unstructured basic log line is not only hard to read, but also makes it more difficult to filter logs by IP, domain or sender for example.
Benefits of structured logging in result queues
Easy to read and understand for applications and systems
With structured logging instead, logging is done in a structured format so that it can easily be parsed and understood by other systems and applications. While there are many different structured log formats, JSON is amongst one of the most commonly used. JSON is a standard format, used and understood by many applications. This makes it easier to read for both humans and machines and requires less parsing.
In MailerQ, we publish JSON result objects to a results queue. Below you find an example of such an JSON object for a failed delivery.
JSON object:
{
"received": "2020-02-04 13:45:14",
"connection": {
"local-ip": "1.2.3.4",
"local-port": 25,
"remote-ip": "5.6.7.8",
"remote-port": 25324,
"secure": true,
"user": "mailerq"
},
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"tags": ["Customer1","Transactional","Invoice"],
"results": [
{
"attempt": 1,
"state": "rcpt to",
"result": "error",
"time": "2020-02-04 13:45:15",
"from": "1.2.3.4",
"to": "10.11.12.13",
"code": 551,
"description": "Mailbox unavailable"
}
],
"extracted": {
"from": "my-sender-address@my-domain.com",
"to": ["info@example.org", "other@example.org"],
"subject": "Test Subject",
"headers": {
"x-example-property": "some-mime-property"
}
}
}They allow you to associate additional properties with keys
Not only are JSON result objects easier to read and understand than the basic log lines, they also offer more information than is practical in the log files. For example message properties that were used when injecting the message, details of the connection over which message was injected, or extracted MIME headers.
While these result properties can also be logged to a traditional basic log file, the JSON property has the added benefit that it is associated with keys. This means that you can see exactly which value corresponds to what property instead of just seeing the values, which allows for faster troubleshooting.
Scalable logging process
In MailerQ, JSON objects are published onto a results queue. This has several additional benefits. The results queue works similar to a publisher/subscriber pattern: publishers (sender of messages) do not directly communicate with the subscribers (receivers). The publisher/subscriber pattern used in the message queues makes the logging process loosely coupled and scalable.
Since publishers and subscribers work independently from each other, it means that if a JSON object fails to get processed from the queue (because it can not be retrieved by the subscribers), other JSON objects can still be published to the queue by the publishers. They will be processed when the system has recovered, so no valuable information gets lost. This is also the case for when there are sudden spikes in volumes.
Integration with monitoring dashboards
The logging setup where JSON result-objects are published to a results queue also makes it easier for senders to analyze and monitor their traffic in overviews or dashboards, especially when done in combination with a log monitoring system such as the ELK stack (Elasticsearch, Logstash, Kibana). The ELK stack is very useful for the indexing of logs, but it can become a bit complicated if you are working with basic log lines. That's where JSON comes in handy.
Because of the structured logging format of the JSON result-object, only minimal parsing is required to integrate ELK for log analysis and nearly all programming languages can parse it. After that you can technically add as many data fields in Kibana with which you can create elaborate dashboards that allow you to quickly identify any issues.
In short
Having extensive and accessible insights into delivery attempts is essential for your email delivery success. While conventional log files can contain a lot of useful information about each attempt, they are difficult to read by both humans as well as machines. Structured JSON logs contain result properties that are associated with keys, so humans and machines can both easily parse them for troubleshooting or statistical purposes. Since MailerQ publishes these JSON result objects to a queue in RabbitMQ, blocking disk I/O is never an issue and the consumers (log monitoring tools such as Logstash) are never dependent on the publisher (the MTA). All in all this allows for extremely rich result data that can be delivered fast and scalably to the right stakeholders, endpoints and tools.
