Returns whether the connection is secure or not. A connection is secure if it has passed the "STARTTLS" state, and all traffic over it is encrypted.
/**
* Get whether the connection is secure or not
*
* @param connection the connection that we want to know the status of
* @return bool
*/
bool MQ_secure(MQ_Connection *connection);
MailerQ offers very fine-grained control over delivery flow, and allows you to pause almost any combination of pool/mta and domain/ip.
A GET request returns all currently active pauses. It returns a JSON array of pauses, see the example below.
GET /v1/pauses HTTP/1.0
Authorization: Bearer ...Might result in
HTTP/1.0 200 Ok
Content-Type: application/json
...
[
{
"mta": "1.2.3.4",
"ip": "123.2.3.4",
"cluster": false
},
{
"mta": "1.2.3.4",
"domain": "gmail.com",
"cluster": false
},
{
"domain": "hotmail.com",
"cluster: true
},
{
"pool": "shared-pool-1",
"domain": "yahoo.com",
"cluster": false
}
]In this example, traffic from 1.2.3.4 to 123.2.3.4 and gmail.com is paused on the instance, all traffic from shared-pool-1 to yahoo.com is paused, and traffic to hotmail.com is paused for the entire cluster.
A POST request allows you to create a new pause, or to update an existing one. For the request format, see the table below. All fields are optional.
| Field | Required | Type | Description |
|---|---|---|---|
| pool | no | string | Pool that the pause applies to. |
| mta | no | string | MTA IP that the pause applies to. |
| domain | no | string | Domain that the pause applies to. |
| ip | no | string | Remote IP that the pause applies to. |
| tag | no | string | The tag that the pause applies to. |
| cluster | no | bool | Whether or not this pause is for the entire cluster or only this instance. |
| expire | no | timestamp | At what time point in the future should this pause be removed |
Do note that not all fields may be passed at the same time. pool and mta are mutually exclusive, and so are domain and ip. That means that they may not be supplied together, or they will result in a 400 response.
For example, the request below will pause sending to hotmail.com for a single campaign.
POST /v1/pauses HTTP/1.0
Content-Type: application/json
Authorization: Bearer ...
{
"domain": "hotmail.com"
"tag": "Campaign1"
}and equivalently, with urlencoded body
POST /v1/pauses HTTP/1.0
Content-Type: application/x-www-form-urlencoded
Authorization: Bearer ...
domain=hotmail.com&tag=Campaign1A DELETE request allows you to remove a pause. For example, the request below removes the pause for the entire cluster to gmail.com from the list of pauses.
DELETE /v1/pauses HTTP/1.0
Authorization: Bearer ...
Content-Type: application/x-www-form-urlencoded
cluster=true&domain=gmail.comThe request format is exactly the same as a POST request, but with the DELETE method. Delete also supports both JSON and urlencoded requests.
Returns whether the connection is considered authenticated. A connection is authenticated if the user has sent a valid username and password, or when a plugin has explicitly marked it as authenticated using the MQ_setAuthenticated() function.
/**
* Get whether the connection is considered authenticated
* @param connection the connection that we want to know the status of
* @return bool
*/
bool MQ_authenticated(MQ_Connection *connection);If your plugin contains this function, it is called right after a message has been sent over an SMTP connection. The function is called after the message was sent, but before the message was accepted.
This function can be useful if you want to write a plugin that monitors which messages were actually sent over the line.
This method is designed to run asynchronously. You should not execute any blocking calls from within this function, if you do you will disturb other active connections resulting in timeouts.
#include <mailerq.h>
/**
* Load a MIME message
*
* @param context Access to the context
* @param connection Access to the SMTP connection
* @param message Access to the JSON message
* @return boolean Do we accept control from here?
*/
MQ_EXPORT bool mq_smtp_out_sent(MQ_Context *context, MQ_Connection *connection, MQ_Message *message) {
// if we return true here, MailerQ will stop processing this message
// until either the MQ_complete, MQ_continue or MQ_retry function is
// called. if none of the functions is called, MailerQ will wait for the
// remote server to close the connection and mark it as an error.
return true;
}
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
MailerQ offers an HTTP injection API. Check out the message format for the required structure of injected messages.
POST /v1/inject HTTP/1.0
Authorization: Bearer ...
Content-Type: application/json
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "From: my-sender-address@my-domain.com\r\nTo: info@example.org\r\nSubject: ..."
}Injected messages simply get published to the inbox queue specified in the config file. The injection endpoint does not support urlencoded post, only JSON directly.
Injected messages will get an extra http property, for its properties see incoming messages.
MailerQ has a non-blocking nature, and uses callbacks to pass control to a plugin. It is then up to the plugin to interact with the event loop, and pass control back to MailerQ when it is ready with its task. Passing back control can be done in three different ways:
The MQ_continue() function can be used for the first of the above options to hand control back to MailerQ. If you call MQ_continue(), MailerQ will call the next plugin in line, or continues with its original algorithm if no other plugins are available.
/**
* Finish plugin processing
*
* @param connection the connection that may move on to the next plugin
*/
void MQ_continue(MQ_Connection *connection);See also MQ_complete() and MQ_retry() for the other ways of handing back control.
This page gives a syntax overview for writing a template that can be used to personalize your mail. Note that although the syntax is quite easy, you can build complicated concepts, which brings the danger of bugs (heck, even simple stuff can have bugs).
Let's just start with the basics of printing a user-supplied variable. A variable
looks like {$name}, {$city} or {$country}: it starts with a {$, followed by
the variable name and is closed with a }. Formally, a variable in personalization:
_);{$NAME} is different from {$name}.The next table gives all variable notations:
| Syntax | Meaning |
|---|---|
{$foo} |
Display a simple variable (no array or object). |
{$foo[4]} |
Display the 5th element of a zero-indexed array. |
{$foo.bar} |
Display the "bar" key value of an object. |
With these notations you may make combinations. Examples of combinations are:
| Syntax | Meaning |
|---|---|
{$foo.bar.baz} |
Display the value behind the key "baz" inside the array "bar" which is a part of $foo. |
{$foo[4].baz} |
Display the value behind the key "baz" inside the 5th element of $foo. |
{$foo.bar.baz[4]} |
Display the 5th element of baz, which is in bar which is in $foo. |
If a variable happens to be an object where the index are keys, you can still access elements in the array with an index number (starting from 0).
While you have to use underscores (_) in your templates when using
variable names, you must supply these in the JSON with dashes (-).
These are converted automatically. Example: When using {$total_price}
in your template, you have to supply this in JSON with "data": { "total-price": 10 }.
Dashes in the template will be interpreted as the subtract operator. Variables with underscores in the JSON won't be accessible in the templates.
You can alter the content of your variables by applying modifiers to them. E.g. you can capitalize the content, calculate the length of a string, or calculating a hash sum. A list of all modifiers and an explanation about their usage can be found here.
Variables containing a numerical value can be used to do some simple math. Just like with normal variables, all math should be done within the {} brackets. So you can for example do the following:
{$var + 10}Besides the surrounding of curly braces, all standard math rules apply.
The standard math operators (+, -, *, /) and the modulo operator (%)
are available. Note that if a value does not exist or does not contain a
numeric value, it will behave as the value zero.
One of the key concepts of any programming language are conditional statements.
A conditional block always starts with the {if} keyword (always with the
curly braces) followed by the statement that is tested. A conditional
block always ends with the if closing tag {/if}. A conditional block is only
executed if the statement in the {if} part is true.
In the next example, the text 'Hello John' is only displayed when the value
of the variable $name is equal to 'john'.
{if $name == 'john'}Hello John{/if}But what if there's also a Sarah in your mailing list. You wouldn't want
to display nothing to her, would you? That's where the {elseif} becomes
the protagonist.
{if $name == 'john'}Hello John{elseif $name == 'sarah'}Hello Sarah{/if}Now, if we want to say something to anyone except to John and Sarah, we will use the {else} keyword. The code after the {else} keyword will be executed if none of the preceeding statements returned true.
{if $name == 'john'}
Hello John
{elseif $name == 'sarah'}
Hello Sarah
{else}
Hello anybody else
{/if}This is of course a very bad example of how you should write your personal salutation because your conditional block would need to be as long as the list of all names in the world.
A much better example would obviously be:
{if $name == ''}
Dear subscriber,
{else}
Dear {$name},
{/if}The code snippet above means: if the value of name is empty, show 'Dear subscriber', otherwise, show Dear John (or Sarah, or what is in the {$name} variable).
In the preceeding example, the operator == was used. This operator means 'is equal to'.
It will evaluate to true if the left hand side of the == is exactly equal
to the right hand side. Operator == is just one of the many operators that
you can use to test the statements.
Here's the complete list:
| Syntax example | Meaning |
|---|---|
| $a == $b | $a and $b are equal |
| $a != $b | $a and $b are not equal |
| $a > $b | $a is greater than $b |
| $a >= $b | $a is greater than or equal to $b |
| $a < $b | $a is less than $b |
| $a <= $b | $a is less than or equal to $b |
| $a =~ "regex" | $a matches with supplied regular expression |
| $a AND $b | $a and $b are true |
| $a OR $b | $a or $b (or both) are true |
| not $a | Negation, will invert the boolean value of $a |
With the first six operators ("==" to "<=") you compare the values of variables $a and $b. If the comparison holds, a true will be returned. If the comparison does not hold, you will get a false. The regular expression operator is special and powerful, and allows you to check if the variable matches a certain pattern.
With the next two operators in the table above (AND and OR) you can combine
statements. If you are writing conditional blocks, you will sometimes find
yourself ending up with blocks that become too long or complex. To shorten
long or more complex conditional blocks, you can put multiple statements in
one single {if} block using the AND and/or OR operators. The AND and/or OR
operators combine the value of $a and $b into a true or a false.
E.g. you can use AND like:
{if $a >= $b AND $b <= $c}true{else}false{/if}In the above snippet first $a >= $b will be tested. Only if this is true,
$b <= $c will be tested. If this happens to be true as well, true will
be printed, else, false will be printed.
Subsequently there is the not operator. This operator will invert the value
that is given. So true becomes false and false becomes true.
Finally it is good to know that a variable itself will be converted to a
boolean when used in an if statement. If the variable contains an empty string
or the value 0, it will be evaluated as false. If it is not an empty string
and it isn't zero, it will be evaluated as true.
If you have a collection of data (an array), and want to see if a specific something is inside that collection, you'll have to loop through all the items in this collection.
Let's use a real world example and display all members of a soccer team,
stored in an array called $soccerTeam.
To get the members of the team, the foreach statement is used. Its syntax is fairly straightforward:
{foreach $player in $soccerTeam}
{$player.name}
{/foreach}This will loop over the items (team members) in $soccerTeam and assign
each player to the variable $player in each iteration. Inside this foreach
block you can do whatever you want with the outputted information. You can
for example generate an HTML list with soccer players from that team.
It becomes a little bit more technical now...
Of course we also support looping over arrays that have non-standard keys.
{foreach $list as $key => $value}
Key {$key} contains {$value}.
{/foreach}As you can see here, you specify both a $key and a $value which will be used to store these values on each iteration.
And finally, sometimes you want certain code to execute if there is no data at all, this is done using the {foreachelse} statement.
{foreach $item in $list}
{$item.name}
{foreachelse}
No items in list.
{/foreach}This foreachelse statement is only executed in case of no data. It is completely ignored otherwise.
It is possible to assign values on runtime. You can for example use this to calculate the total price of a set of purchased items. Or to remember a certain item inside a foreach statement. Assigning variables is done as follows:
{assign $item to $topitem}After this statement the variable $topitem is available and it will contain what $item contained when the email was being compiled and sent to the user.
This will allow you to do things like the following.
{foreach $item in $list}
{assign $total + $item.price to $total}
{if $item.price > $topitem.price}
{assign $item to $topitem}
{/if}
{/foreach}Which will eventually have the most expensive item in the $topitem variable
and the total price in $total.
As you can see above the syntax heavily depends on the curly braces {}.
Therefore, these braces cannot appear in the text without leading to a
valid syntax. If you do need these braces in you text, you can use: {ldelim}
to get a { and {rdelim} to get a }. If you have a large text with
these braces you can enclose the text in {literal} and {/literal}.
This text will then not be processed.
The code snippets above are formatted in a way to make them readable: we used newlines, indentation, etc. When you are generating text for the html part of your mail this is advisable since it is easier to capture mistakes. However, if you are generating content for the text part of your mail, you should be aware that the formatting of your template code affects the format of your text.
E.g. above we gave a snippet that prints out names of players in a soccer team
If the soccerTeam array looks like: ["Ronaldo", "Messi", "Ibrahimovic"]
The output that we get in the text section will be:
Ronaldo
Messi
Ibrahimovic
This is probably not what you had in mind while typing the foreach loop. But this is what you get since there is a new line before and after the variable and there are some spaces in front of it used as indentation. To get a list of the names without the extra whitespace you should write:
{foreach $player in $soccerTeam}{$player}
{/foreach}This is less readable but gives what you want
Ronaldo
Messi
IbrahimovicEnvelope and recipient addresses are wrapped in MQ_Address structures. You can get a reference to this structure by calling the MQ_envelope() or MQ_recipient() function on a MQ_Message structure, or by implementing the mq_smtp_in_mailfrom() or mq_smtp_in_rcptto() functions.
Once you have access to a MQ_Address struct, you can call the following functions on it to get access to the underlying address, and to other address properties.
| Callable function | Description |
|---|---|
| MQ_email() | Access to the underlying email address |
| MQ_local() | Check whether a recipient address is "local" |
| MQ_setLocal() | Mark a recipient as "local" |
This function can be called to retrieve user data set earlier with MQ_setContextData.
Every plugin may associate data with a plugin. This data pointer is completely ignored by MailerQ, but allows plugins to store data between different callbacks for the same context. The data pointer is unique per plugin, so you do not have to worry that this function returns pointer to data that is set by a different plugin.
If no data was set earlier, this function returns NULL.
/**
* Retrieve previously stored data.
*
* @param context the context to retrieve the data from
* @return void*
*/
void *MQ_contextData(MQ_Context *context);This function will stop monitoring a file descriptor.
/**
* Stop monitoring a file descriptor
*
* @param watcher the watcher to deactivate
*/
void MQ_ioUnwatch(MQ_IOWatcher *watcher);For a more detailed example, see the MQ_ioWatch documentation
If you want to retrieve the full email address stored inside the MQ_Address structure you can use this function.
The pointer that is returned should not be freed by the user. MailerQ will automatically free it when it is done processing the message.
/**
* Retrieve the email address given an envelope structure
*
* Be aware: This function may return a nullptr in case no
* envelope address exists. This is most commonly
* seen in DSN messages.
*
* @param envelope the envelope address
* @return const char *
*/
const char *MQ_email(MQ_Address *address);This function is called by MailerQ right before a worker thread is stopped and destroyed. You can for example use it to deallocate memory that you had earlier allocated.
This function can be implemented by the plugin and is called by MailerQ when a worker thread is about to stop. It is not possible to take control by returning boolean true here, but since this function is called during cleanup it is acceptable to block here.
#include <mailerq.h>
/**
* This function is called by MailerQ for every worker thread
* that comes to an end
*/
MQ_EXPORT void mq_context_cleanup(MQ_Context *context) {
// get the mysql connection resource that we stored in the context
MYSQL *connection = (MYSQL *)MQ_ContextData(context);
// close the connection
mysql_close(connection);
// TODO: add other useful cleanup stuff
}MailerQ is compatible with a lot of languages. For a language to be compatible with the MTA, all that is needed is support for JSON and the language being able to communicate with RabbitMQ. Some of the languages where this is possible are: C#, Erlang, Java, .NET, Perl, PHP, Python, Phyton-Puka and Ruby.
For the examples below, it is asumed that MailerQ will be able to communicate with the RabbitMQ and MySQL servers.
This example assumes that MailerQ is running and configured. It uses the Java client library provided by RabbitMQ.
The Java example consists of two files; Send.java and Result.java. If you want
to use these scripts, save the two files inside the same directory.
SendThis class connects to RabbitMQ and places a JSON encoded message on the outbox
message queue. To connect to RabbitMQ and to create the JSON encoded message,
the program uses the private variables defined inside the class. If you want
to use the Java example to test your MailerQ configuration, you will have to run
the Send class before running the Result class.
Send.java// import the rabbitmq libraries that are needed to connect to RabbitMQ
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
// import the rabbitmq lbrary that is needed to create the RabbitMQ channel
import com.rabbitmq.client.Channel;
/**
* Simple Java class that creates a connection to a RabbitMQ server and
* places a message on the queue, so MailerQ can take this message from
* the queue and send it out as an email.
*/
public class Send {
/**
* Settings for RabbitMQ
*/
// same as the rabbitmq-host option in the config file
private static String hostname = "localhost";
// same as the rabbitmq-user option in the config file
private static String username = "guest";
// same as the rabbitmq-password option in the config file
private static String password = "guest";
// same as the rabbitmq-vhost option in the config file
private static String vhost = "/";
// same as the rabbitmq-outbox option in the config file
private static String outbox = "outbox_test";
/**
* Settings for sending out the test email
*/
// domain where the test message should be delivered
private static String recipientDomain = "example.org";
// email where the test message should be delivered
private static String recipientEmail = "info@example.org";
// address where the email was sent from
private static String fromAddress = "me@my-domain.com";
/**
* Main method
*/
public static void main(String[] argv) throws Exception {
// create a new ConnectionFactory
ConnectionFactory factory = new ConnectionFactory();
// set the host to RabbitMQ
factory.setHost(hostname);
// set the user name to connect to RabbitMQ
factory.setUsername(username);
// set the password to connect to RabbitMQ
factory.setPassword(password);
// set the virtual host to RabbitMQ
factory.setVirtualHost(vhost);
// create the new connection
Connection connection = factory.newConnection();
// create the channel
Channel channel = connection.createChannel();
// declare the queue
channel.queueDeclare(outbox, true, false, false, null);
// create the message
String message = "{ "\"envelope\": \"" + fromAddress + "\", "
+ "\"recipient\": \"" + recipientEmail + "\", "
+ "\"mime\": \"" + "From: " + fromAddress + "\r\nTo: "
+ recipientEmail
+ "\r\nSubject: Example subject\r\n\r\n"
+ "This is the example message text\" }";
// publish the message on the queue
channel.basicPublish("exchange1", "key1", null, message.getBytes());
// some output
System.out.println("Sent: '" + message + "'");
// close the channel
channel.close();
// close the connection
connection.close();
}
}This class connects to the RabbitMQ server and gets the messages that were
placed on the outbox message queue by the Send class back from the result
message queue. The result messages from the result queue are shown to the user.
The Result class can only output any relevant information when it is executed
after the Send class was run. The Result program will keep running, waiting for
new messages until it is terminated (use Ctrl-C). You could run the Send class
from another terminal, to test the effect.
Result.java// import IOException
import java.io.IOException;
// import the rabbitmq libraries that are needed to connect to RabbitMQ
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;
// import the rabbitmq lbrary that is needed to create the RabbitMQ channel
import com.rabbitmq.client.Channel;
// import the rabbitmq library that can be used to buffer the message that
// was pushed by the RabbitMQ server
import com.rabbitmq.client.QueueingConsumer;
public class Result
{
/**
* Settings for RabbitMQ
*/
// same as the rabbitmq-host option in the config file
private static String hostname = "localhost";
// same as the rabbitmq-user option in the config file
private static String username = "guest";
// same as the rabbitmq-password option in the config file
private static String password = "guest";
// same as the rabbitmq-vhost option in the config file
private static String vhost = "/";
// same as the rabbitmq-results option in the config file
private static String resultbox = "results_test";
public static void main(String[] argv)
throws java.io.IOException, java.lang.InterruptedException
{
// create a new ConnectionFactory
ConnectionFactory factory = new ConnectionFactory();
// set the host to RabbitMQ
factory.setHost(hostname);
// set the user name to connect to RabbitMQ
factory.setUsername(username);
// set the password to connect to RabbitMQ
factory.setPassword(password);
// set the virtual host to RabbitMQ
factory.setVirtualHost(vhost);
// create the new connection
Connection connection = factory.newConnection();
// create the channel
Channel channel = connection.createChannel();
// declare the queue
channel.queueDeclare(resultbox, true, false, false, null);
// message telling the user that the script is waiting for messages
// that will be placed on the result queue by MailerQ
System.out.println("Waiting for messages to be placed on the result queue. To exit press CTRL+C");
// create the QueueingConsumer
QueueingConsumer consumer = new QueueingConsumer(channel);
// start the consumer
channel.basicConsume(resultbox, true, consumer);
// keep waiting for messages
while (true)
{
// get a delivery from the result queue
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
// get the message as a string
String message = new String(delivery.getBody());
// print a message from the queue on the screen
System.out.println("Received:");
System.out.println(message);
System.out.println("------------------------------------------");
}
}
}You can compile both of these classes with just the RabbitMQ client on the classpath.
$ javac -cp rabbitmq-client.jar Send.java Result.java
To run the the Sender class
$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar Send
To run the the Result class
$ java -cp .:commons-io-1.2.jar:commons-cli-1.1.jar:rabbitmq-client.jar Result
Function that can be added to a plugin that is called every time a "MAIL FROM" instruction is received by MailerQ.
bool mq_smtp_in_mailfrom(MQ_Context *context, MQ_Connection *connection, MQ_Address *envelope);
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
MailerQ comes equipped with a full MTA management console. The management console allows you to monitor the performance of your email delivery in real-time. If necessary, the management console can be used to change settings to maximize deliverability on the fly. If you have multiple instances of MailerQ running in a cluster, you can easily switch between them.
The management console can be enabled in MailerQ's configuration file. The following variables should be used:
www-port: 8485 (default: 8485)
www-ip: 1.2.3.4 (default: 0.0.0.0, meaning all available IP's)
www-dir: /usr/share/mailerq/www (default: /usr/share/mailerq/current/www)
www-connections: 10The www-port variable holds the port number for the management console;
8485 is the default. If you use port 80 (which is the default port for HTTP
traffic) you can access the management console with using a browser via
address http://hostname.of.your.server. If you assign a different port number
(like 8485), you must include the port number in the URL: http://hostname.of.your.server:8080.
In its default setting of 0.0.0.0, the management console is accessible via
all IP addresses that are assigned to the server on which MailerQ runs. If you
only want to make it accessible via one specific IP, you can set the www-ip
variable. Of course, the IP address that you assign must be bound to the server.
The management console is protected with a username and password to prevent anyone from accessing it. Besides setting a password, we also recommend to put the management console behind a firewall so that you will not have to worry about people breaking into it.
Since MailerQ 5.12, there are many possible ways to let users authenticate to the management console. LDAP, RADIUS, linux users, HTTP endpoints, username/password files and lastly custom scripts.
www-auth: hardcoded://username:password,username2,password2
www-auth: ldap://example.com:389/base_dn
www-auth: radius://identifier:secret@example.com:1812
www-auth: users://user1,mailerq,user2,client*
www-auth: http://example.com:80/some_endpoint
www-auth: file:///users.txt
www-auth: exec:///path/to/fileThe first hardcoded is simply a comma separated list of username:password pairs, separated
by the ':' character. Note that this is the least safe, since the passwords will be plainly
readable in the config file, in unhashed form.
The ldap specifier will connect to the LDAP server and try to bind to DN=uid=username,base_dn.
If this succeeds, the user is authenticated, if this fails, the user is rejected.
The radius specified will simply request access to the radius server, with the shared secret secret and the identifier identifier. Make sure that the
client is set up correctly, or the server will reject the packets and MailerQ
logins will fail.
The http will perform HTTP BASIC authentication to the endpoint, and will therefore send along the
Authorization: Basic ... header. Any 2xx code is treated as a success, while any other code is
treated as a failure and a rejection.
The file specifier will load a simple text file that contains the usernames and passwords separated
by the : character, but with the passwords in hashed form (similar to /etc/passwd). These passwords
can be generated using mkpasswd -m sha-512 or openssl passwd -6, for example.
Lastly, the exec specifier will run the /path/to/file in a subprocess, and will check the return
code. This way, you can easily implement any authentication you require in a script! For example, you
can check a username and password combination in an SQL database, or you can ues the linux PAM modules,
or you can make very custom requests. This allows you to tie the management console authentication
directly into your existing user management infrastructure.
All HTML, CSS and Javascripts that are necessary for the management
console are automatically installed into the /usr/share/mailerq/current/www
directory. If you want to run the console from out of a different
location, you can change this directory with the www-dir variable.
To limit the number of resources that can be used by the built-in HTTP server, you can use the "www-connections" variable to limit the number of simultaneous HTTP connections that can be handled. This number includes active web sockets.
If is a good idea to secure your management console, as it will also be used to manage private DKIM Keys; by definition, these should be kept private and thus not transferered over interceptable non-secure HTTP connections.
The following configuration file variables are relevant for enabling HTTPS support:
www-secure-port: 443 (empty by default)
www-certificate: /path/to/certificate.crt (empty by default)
www-privatekey: /path/to/privatekey.key (empty by default)
www-ciphers: !aNULL:!eNULL:!LOW:!SSLv2:!EXPORT:!EXPORT56:FIPS:MEDIUM:HIGH:@STRENGTH (empty by default)If you enable both HTTP and HTTPS, users who access the non-secure interface
will automatically be forwarded to the secure interface. The www-secure-port
holds the port number for the HTTPS connections (443 is the default for
this, so that you won't have to include the port number in URLS). The
certifate and key files, and the supported ciphers can be set using
the www-certificate, www-privatekey and www-ciphers variables.
Once enabled, the encrypted management console can be accessed using
the address https://hostname.of.your.server if you use default port 443,
or https://hostname.of.your.server:port for any other port.
If you have a cluster with multiple MailerQ instances, the web interface of these interfaces contains links to the other instances. MailerQ does its best to find out the URL for each of the other interfaces (by combining the host names and port numbers), but you can use the following optional config file variables to help a hand:
www-host: your.hostname (default: auto-detected)
www-url: https://your.hostname:port (default: auto-detected)The management console allows administrators to monitor live SMTP traffic. All incoming or outgoing connections can be intercepted, and the entire SMTP handshake (EHLO, MAIL FROM, RCPT TO, DATA) is real time visible via the management console. You can thus see he raw MIME message data that is being sent or received.
In fact, the management console even has an option to not only display the raw MIME data, but to extract the HTML source code too, and render this in your browser. With this tool you can exactly see what type of messages your users are sending, and you can take action if you see messages that look like spam, phishing or other types of abuse.
However, if you use the console to look at rendered emails, your browser automatically downloads images and other resources from the mail as well. This could trigger actions (like statistics updates) on the servers where these files are hosted. If you do not want this and you have control over these servers, you can take precautionary measures, for example by ignoring downloads that come from your IP address, or by ignoring downloads from clients with a specific user agent setting.
At Copernica we use this approach. Our devops have browsers with a special user agent setting (with Firefox you can change this using the "about:config" url), so that clicks and opens from us do not pollute mailing stats. In MailerQ's config file we have set the "render-useragent" value too, so that MailerQ refuses to share emails with browsers with a different user agent setting.
render-useragent: Copernica DevOpsThe "render-useragent" setting ensures that only browsers with the specified user agent can render emails. If you have also updated your tracking servers to ignore clicks and opens from browsers with this user agent setting, you can make sure that you can safely look at rendered emails on the management console, without triggering any actions on your tracking servers.
The resources for the web interface (like the html, css, and javascript files) are sometimes cached by your browser. This is normally ok (because you probably do not update the files anyway), but if you plan to install a new MailerQ release once every while, you can set the cache-control header in the config file:
www-cache-control: must-revalidateThis config file option was mainly added to MailerQ to help the MailerQ web developers: they make changes to the interface all the time, and they wanted to prevent that the testers were reviewing an outdated interface. But in production environments, this setting might be useful too.
Check out the documentation on mozilla.org for a list of supported options for this header.
If you want to get access to the JSON data that was loaded from RabbitMQ (for plugins that interact with messages that are going to be sent) and/or with the JSON data that is going to be published to RabbitMQ (for plugins that want to modify incoming messages), you can call the MQ_json() function.
This function returns a "struct fjson_object". This is a pointer to a data from the >a href="https://github.com/rsyslog/libfastjson">libfastjson library. If you want to work with this data pointer, you therefore also need access to this JSON handling library.
/**
* Retrieve JSON data from a message
*
* @param message the message to retrieve the JSON from
* @return json pointer to the JSON object
*/
struct fjson_object *MQ_json(MQ_Message *message);
For more information on how to work with the JSON structure, please check the documentation of the libjson library:
This function starts a timer. When the given time runs out, the callback will be invoked, unless the timer is stopped first with MQ_stopTimer.
/**
* Start a timer
*
* @param context the context to start the timer in
* @param timeout the number of seconds to wait before invoking the callback
* @param callback the callback to invoke when the timer runs out
* @param data customer user data to provide to the callback
*/
MQ_Timer *MQ_startTimer(MQ_Context *context, float timeout, void(*callback)(void *data), void *data);
Using timer functionality, we can amend the example for the MQ_ioWatch function so that it times out when no input is entered for at least five seconds.
#include <mailerq.h>
#include <string.h>
#include <stdio.h>
/**
* Structure that references both timers
*/
struct Timers
{
/**
* The IO watcher
*/
MQ_IOWatcher *io;
/**
* The timer
*/
MQ_Timer *timer;
};
/**
* Callback function to receive standard input
*
* @param watcher the watcher that is monitoring the file descriptor
* @param fd the file descriptor that received events
* @param data custom data pointer
*/
void io_callback(MQ_IOWatcher *watcher, int fd, int events, void *data)
{
// buffer for data to process
char data[1024];
ssize_t length;
// retrieve the timers
struct Timers *timers = (struct Timers*)data;
// we are working with STDIN, which is never writable
// (because that's what we have STDOUT for), so we don't
// have to check the events variable, otherwise we could
// check for readability or writability like this:
//
// events & MQ_READ
// events & MQ_WRITE
// normally, when we check for readability we should not
// assume that we are receiving complete lines, however
// STDIN is line-buffered, so the callback is only invoked
// whenever the user presses return after input
// read data from input
length = read(fd, data, sizeof(data));
// if there are only two characters we received an empty line
// since the line includes the CRLF produced by the terminal
if (length == 2)
{
// stop watching the file descriptor
MQ_ioUnwatch(watcher);
// deactive the timer
MQ_stopTimer(timers->timer);
// clean up the timers
free(timers);
}
else
{
// reset the timer to five seconds
MQ_resetTimer(timers->timer, 5.0);
}
}
/**
* Callback function for an expiring timer
*
* @param data custom user data
*/
void timer_callback(void *data)
{
// get the timers
struct Timers *timers = (struct Timers*)data;
// we did not receive any input for 5 seconds
// so we stop listening to input
MQ_ioUnwatch(timers->io);
// and free the timers
free(timers);
}
/**
* Monitor standard input for data
*
* @param context the context to monitor in
*/
void monitor(MQ_Context *context)
{
// create the structure holding the timers
Timers *timers = malloc(sizeof(struct Timers));
// set the context
timers->context = context;
// monitor stdin for input
timers.io = MQ_ioWatch(context, STDIN, MQ_READ, callback, timers);
// and stop monitoring after five seconds
timers.timer = MQ_startTimer(context, 5.0, timer_callback, timers);
}
MailerQ can monitor a directory on the file system for new files: the spool directory. You can simply inject email into MailerQ by copying files to this directory.
The files that you drop in the spool directory must be correctly formatted MIME messages. Invalid files will not be picked up. MailerQ opens the file, and filters out all the recipients from the "to", "cc" and "bcc" header fields. For each recipient a JSON formatted message is copied to the "inbox" queue (set with the "rabbitmq-inbox" variable).
The envelope address is also extracted from the message: the "x-mq-envelope" header is preferred. If this header is missing, the "return-path", "from" and "sender" headers are checked; the first valid email address that is recognized in these headers will be used as the envelope address.
The mail is published to the "inbox" queue. If you've assigned the outbox queue to the "rabbitmq-inbox" setting, the email is immediately delivered to the recipient(s).
Although using the spool directory is very convenient for sending out mails it is also very slow. This is caused by the fact that every mail needs to be written to and read from disk. We therefore encourage you to use the other methods to send mails to MailerQ.
In the config file the following options are relevant for the spool directory:
spool-directory: /path/to/directory
spool-fail-directory /path/to/directory
spool-delay: 0 (default: 0)
spool-remove: true (default: true)
spool-extract: true (default: true)
spool-open-files: 10 (default: 10)
spool-threads: 1 (default: 1)The "spool-directory" is the most important one. It contains the path to the directory that should be monitored. All files that you copy to this directory are immediately picked up, processed and removed. This happens in a blink of an eye. If you need a small delay before messages are picked up, you can use the "spool-delay" variable. Set this to a numeric value holding the number of seconds to wait before files are picked up.
After publishing the message to the inbox queue, MailerQ removes the input file. If you do not want this, you can set "spool-remove" to false. However, it is not recommended to disable removing files, because emails might get redelivered if MailerQ is restarted and starts scanning the spool directory again.
If the contents of a file cannot be processed, MailerQ can move the file to another directory. The path of the directory can be set using the "spool-fail-directory" option. If this setting is left empty, files are not moved from the spool directory.
All mails that you drop in the spool directory are scanned for headers that start with "x-mq-*". Such headers are filtered out and converted to JSON properties that control the delivery of the mail. If you do not want to scan for such headers, you can use the "spool-extract" variable to disable this.
MailerQ supports processing multiple files in the spool directory simultaneously. The variable "spool-open-files" sets the maximum number of files MailerQ has open at any given time. To use more threads for processing files, you can set "spool-threads".
Function that can be implemented by a plugin, and that is called by MailerQ when an incoming SMTP connection is closed. This is the perfect moment for a plugin to run cleanup code, for example to deallocate memory that is assigned to the connection with the MQ_setConnectionData() function.
bool mq_smtp_in_close(MQ_Context *context, MQ_Connection *connection);
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
This function can be called to store data which can then be retrieved later using the MQ_contextData() function. The data is segmented per plugin, so data stored will not overwrite data stored by other plugins, or in different contexts.
/**
* Store data in the context
*
* @param context the context to store the data in
* @param value the data to store
*/
void MQ_setContextData(MQ_Context *context, void *value);Copernica, the company that develops MailerQ, also operates an online service to create responsive emails based on JSON input: responsiveemail.com. The algorithm to transform JSON into responsive emails has been embedded in MailerQ. If you feed MailerQ JSON (instead of MIME), it will be converted into valid email messages.
{
"envelope": "sender@example.com",
"recipient": "receiver@example.com",
"mime": {
"from": {
"name": "John Doe",
"address": "sender@example.com",
},
"subject": "Example email",
"text": "Text version",
"background": {
"color": "white"
},
"content": {
"blocks": [{
"type": "html",
"content": "<p>Example content</p>"
}, {
"type": "button",
"label": "click here"
}]
}
}
}MailerQ reads JSON objects from the RabbitMQ "outbox" queue. These JSON objects hold properties like "envelope", "recipient" and "mime", where the "mime" property is normally set to a string holding the full message source in MIME format. However, you can also assign a nested JSON object to this "mime" property (see above example). This nested object is processed by MailerQ and transformed into a MIME message.
There are many nested properties that are supported by the algorithm, which are all documented on the ResponsiveEmail.com website. Note that the properties that are described as top-level properties on the responsiveemail.com website, are (of course) nested properties inside the "mime" property of the object that is processed by MailerQ.
The ResponsiveEmail algorithm reads in a JSON object in which the texts, buttons and images are defined, and generates the HTML and MIME content based on that input. This means that you do not have to write the message source (HTML, CSS and MIME) yourself, or worry about the tricks to make your message compatible with all the various email clients.
However, if you do want to take care of the HTML yourself, you can also use the ResponsiveEmail algorithm for just the HTML-to-MIME conversion. In that case you just specify the header properties (from and to address, subject line) and the HTML and text source of your message, and rely on MailerQ to turn this into a valid MIME message:
{
"envelope": "sender@example.com",
"recipient": "receiver@example.com",
"mime": {
"from": {
"name": "John Doe",
"address": "sender@example.com",
},
"to": {
"name": "Mister Receiver",
"address": "receiver@example.com",
},
"cc": {
"name": "Someone Else",
"address": "randomguy@example.com",
},
"subject": "Example email",
"headers": {
"list-unsubscribe": "<mailto:unsubscribe@example.com>"
},
"text": "This is the pure text version of the email",
"content": "<html><head> ..... </body></html>",
"attachments": [ {
"url": "http://www.example.com/document.pdf"
} ]
}
}If you use the full ResponsiveEmail algorithm (where you do not have to write the HTML code yourself), you assign a deeply nested JSON object to the "content" property. Inside this "content" object you specify the images, texts, buttons and other elements that you want to include in your email. MailerQ turns this content into a responsive email message. However, in the above example we assigned a string to the "content" property. By doing this, MailerQ creates a MIME property with exactly the text and HTML that you've set in the JSON.
When emails are generated, MailerQ sometimes has to download resources from the internet to find out the dimensions of images and to fetch attachments. To prevent that the same resources are downloaded over and over again, MailerQ can be configured to use a cache. In the configuration file you can set the address of this cache:
download-cache: directory:///path/to/dir
download-ttl: 3600The "download-cache" property holds the address of the cache. The format of this address is the same as the message-store-options variable, and can refer to a directory (like we did above) or a nosql database if you use an address that starts with "mongodb://" or "couchbase://".
MailerQ inspects the 'cache-control' and 'expires' headers in the HTTP response to decide for how long a resource can be stored in the cache. With the 'download-ttl' config file variable you can set the upper limit for this. If you set it to 3600, downloaded files will stay for at most one hour (3600 seconds) in the cache, even if the HTTP response headers allowed a longer cache time.
For the downloads, a custom user agent can be supplied. By default, the user agent
mailerq uses is 'MailerQ
download-user-agent: My custom user agentMany users run MailerQ on a server with unrestricted access to the local network. MailerQ is then not only capable of fetching resources from the internet, but also directly from servers inside this local network. This makes it, for example, possible to use attachment URL's like "http://192.168.0.22/attachment.pdf". This could be a security risk, especially if the attachment and image URL's in your mails can be entered by third parties.
To prevent such internal downloads, you can use the config file variables "download-blacklist" and "download-whitelist" to limit the IP addresses to which MailerQ can connect.
download-blacklist: 192.168.0.0/16,127.0.0.1
download-whitelist: 192.168.0.30/32:80Downloads from IP address on the blacklist are forbidden, unless the IP is also included in the whitelist. The above configuration prevents that MailerQ can connect to servers on the local network, with an exception for port 80 of IP 192.168.0.30.
It is possible that the server cannot access the internet, except through a proxy. That
is why there is a config option to set up either a SOCKS or an HTTP proxy for downloads.
For the proxy URI format, see proxy settings in other options.
If unset, the global proxy-server variable will be the fallback.
download-proxy: <proxy URI> (empty by default)The AmqpDelayer is an application designed to manage delayed message handling within RabbitMQ message queues. It reads messages from a RabbitMQ queue and, based on the "timestamp" header property, holds each message until its designated delay expires. When this delay expires, AmqpDelayer publishes the message back to RabbitMQ for further processing. Messages without a timestamp, or those marked for immediate release, are forwarded right away. Storage can be configured as disk-based or use MongoDB.
We developed this application for MailerQ users seeking alternatives to prolonged message storage in RabbitMQ. When an email is rescheduled for delayed delivery, such as due to temporarily unreachable mail servers or graylisting, MailerQ returns it to RabbitMQ, where it is potentially held for up to 48 hours. Since RabbitMQ isn’t optimized for such long-term storage, AmqpDelayer can serve as an optional intermediary, offloading these retry messages from RabbitMQ and retaining them on disk or in MongoDB until they’re ready for re-queueing.
Although the application was originally developed for use in combination with MailerQ, it is a generic application that can be used in all sorts of RabbitMQ based topologies where messages need to be offloaded from RabbitMQ for a while.
The application is stored in our APT repository. If you have already enabled our APT repository, you can install it with this command:
sudo apt install mailerq-amqpdelayerA typical use case is to route the MailerQ retries to a special queue, and configure AmqpDelayer to read out this queue, and route the output back to the outbox queue, where it will be picked up by MailerQ again for further delivery.
You typically run this program from the command line. It reads all its input from a config file stored in /etc/copernica/amqpdelayer.txt. However, you can also pass command line arguments, for example:
amqpdelayer --rabbitmq-address amqp://user:password/hostname/vhostAll options can be supplied in the system-wide config file, via environment variables, and as command line options. The config file is stored in /etc/copernica/amqpdelayer.txt. Each config file setting can also be passed as a command line argument, or supplied as an environment variable (where the variable name is converted to uppercase, and dashes to underscores). The following settings are thus equivalent:
All other configuration settings can be converted in a similar manner.
AmqpDelayer reads messages from a RabbitMQ message queue, holds them for a while, and then publishes them back to RabbitMQ. The following options can be used to configure this:
If you leave the routing key empty, messages will be published to the exchange with their original routing key.
There are two types of storage supported: disk-space storage and MongoDB-based storage.
Disk-based storage is implemented using LMDB and uses separate data and index files. The mapsize config file setting holds the max size for such files. It is not used for MongoDB-based storage.
AmqpDelayer does not use locks or other mechanisms to prevent data races: if you run multiple AmqpDelayer instances, each one must have its own directory or MongoDB collection where data is stored.
Messages are delayed based on the timestamp header in the message. When it is set to a future timestamp, messages are stored. Other messages are immediately published back to RabbitMQ. The following settings can be used to further control this:
These settings can be used to prevent that messages that expire soon are sent to storage. All values are in seconds.
The application normally takes care of both delaying and reviving messages. With the mode setting, you can configure the application to just delay messages (without ever reviving them), or to just revive them (without ever delaying):
When the application shuts down, messages are kept in storage and are picked up again when the application is restarted. You can alternatively set the shutdown procedure to "return" or "forward". With "return" all stored messages are published back to the queue from which they were read, and with "forward" the messages are passed to the configured exchange and routingkey as if they already expired.
The shutdown option could be helpful when running in a containerized environment with non-persistent storage (like emptyDir on Kubernetes). Messages are published back to RabbitMQ when the application closes down and storage is removed. The shutdown procedure is not implemented for MongoDB-based storage.
AmqpDelayer uses several settings to improve throughput for incoming messages and publishing to RabbitMQ. Below are key configuration options to tune RabbitMQ quality of service and storage operations.
Higher QoS values allow AmqpDelayer to load multiple messages into memory, bundling them into write transactions. QoS should always be set higher than the group size to ensure efficient bundling. Message are acked to RabbitMQ after they are written to storage.
The group size sets the write transaction’s size. A smaller group size keeps messages in memory for less time, while a larger group size reduces write frequency. Ensure QoS is set higher than the group size for effective message buffering.
To avoid overwhelming RabbitMQ when republishing messages, use these settings:
The throttle prevents AmqpDelayer from sending too many messages to RabbitMQ simultaneously. Once the throttle limit is hit, AmqpDelayer pauses loading from storage until at least batchsize messages are acknowledged. This ensures a steady message flow without overloading RabbitMQ. Batchsize must be smaller than the throttle.
The application has a small built-in web server that exposes some metrics that
can be used in a Prometheus stack. There are two supported pages: / and /metrics.
Function that can be added to a plugin that is called every time a "RCPT TO" instruction is received by MailerQ.
bool mq_smtp_in_rcptto(MQ_Context *context, MQ_Connection *connection, MQ_Address *address);
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
The list below is a standard list with the SMTP reply codes according to the SMTP standard. Beware however that this is just a standard, and that certain SMTP servers might return their own status, codes and descriptions that differ from the list below.
| Code | Status (SMTP Error) | Description |
|---|---|---|
| 211 | 2.1.1 | System status, or system help reply |
| 214 | 2.1.4 | Help message |
| 220 | 2.2.0 | |
| 221 | 2.2.1 | |
| 250 | 2.5.0 | Requested mail action okay, completed |
| 251 | 2.5.1 | User not local; will forward to <forward-path> |
| 354 | 3.5.4 | Start mail input; end with <CRLF>.<CRLF> |
| 421 | 4.2.1 | <domain> Service not available, closing transmission channel |
| 450 | 4.5.0 | Requested mail action not taken: mailbox unavailable |
| 451 | 4.5.1 | Requested action aborted: local error in processing |
| 452 | 4.5.2 | Requested action not taken: insufficient system storage |
| 500 | 5.0.0 | Syntax error, command unrecognised |
| 501 | 5.0.1 | Syntax error in parameters or arguments |
| 502 | 5.0.2 | Command not implemented |
| 503 | 5.0.3 | Bad sequence of commands |
| 504 | 5.0.4 | Command parameter not implemented |
| 521 | 5.2.1 | <domain> does not accept mail |
| 530 | 5.3.0 | Access denied |
| 550 | 5.5.0 | Requested action not taken: mailbox unavailable |
| 551 | 5.5.1 | User not local; please try <forward-path> |
| 552 | 5.5.2 | Requested mail action aborted: exceeded storage allocation |
| 553 | 5.5.3 | Requested action not taken: mailbox name not allowed |
| 554 | 5.5.4 | Transaction failed |
MailerQ offers very fine-grained control over delivery flow, and allows you to intercept it and forcing an error on any combination of pool/mta and domain.
A GET request returns all currently active errors. It returns a JSON array of errors, see the example below.
GET /v1/errors HTTP/1.0
Authorization: Bearer ...Might result in
HTTP/1.0 200 Ok
Content-Type: application/json
...
[
{
"mta": "1.2.3.4",
"ip": "123.2.3.4",
"cluster": false,
"code": 500
},
{
"mta": "1.2.3.4",
"domain": "gmail.com",
"cluster": false,
"code": 421,
"description": "forced deferring"
},
{
"domain": "hotmail.com",
"cluster: true,
"code": 521,
"extended": "5.2.1"
},
{
"pool": "shared-pool-1",
"domain": "yaho0.com",
"cluster": false,
"code": "553",
"description": "invalid mailbox"
}
]In this example, traffic from 1.2.3.4 to 123.2.3.4 is failed with an error code 500, anc traffic to gmail.com is deferred with
an error code of 421 to be tried again later. In the third value, all emails to hotmail.com get an 521 error on the entire cluster.
Lastly, all mail from shared-pool-1 to yaho0.com on this instance are failed because the mailbox is invalid.
A POST request allows you to create a new error, or to update an existing one. For the request format, see the table below. All fields are optional.
| Field | Required | Type | Description |
|---|---|---|---|
| code | yes | int | Numeric error code between 200 and 599 (smtp error codes). |
| extended | false | string | Extended SMTP error code, e.g. 5.7.1. |
| description | false | string | Description to put in the message result. |
| pool | no | string | Pool that the error applies to. |
| mta | no | string | MTA IP that the error applies to. |
| domain | no | string | Domain that the error applies to. |
| tag | no | string | The tag that the error applies to. |
| cluster | no | bool | Whether or not this error is for the entire cluster or only this instance. |
Do note that not all fields may be passed at the same time. pool and mta are mutually exclusive.
That means that they may not be supplied together, or they will result in a 400 response.
For example, the request below will install a Forced Error with code 421 to hotmail.com for a single campaign.
POST /v1/errors HTTP/1.0
Content-Type: application/json
Authorization: Bearer ...
{
"domain": "hotmail.com",
"code": 421,
"tag": "Campaign1"
}and equivalently, with urlencoded body
POST /v1/errors HTTP/1.0
Content-Type: application/x-www-form-urlencoded
Authorization: Bearer ...
domain=hotmail.com&code=421&tag=Campaign1A DELETE request allows you to remove an error. For example, the request below removes the error for the entire cluster to gmail.com from the list of errors.
DELETE /v1/errors HTTP/1.0
Authorization: Bearer ...
Content-Type: application/x-www-form-urlencoded
cluster=true&code=421&domain=gmail.comThe request format is exactly the same as a POST request, but with the DELETE method. Delete also supports both JSON and urlencoded requests.
If you want to retrieve the MIME data from a MQ_Message, you can use this function. It returns a NULL terminated string, or simply NULL if no mime has been set.
/**
* Retrieve the MIME data
*
* @param message the message to retrieve the mime from
* @return the MIME data
*/
const char *MQ_mime(MQ_Message *message);This function has a different behavior if you call it on a message that is being received and on a message that is being sent. If you call it on a message that is being received, for example inside your mq_smtp_in_message() function, it is simply an alias for retrieving the "mime" property from the JSON.
For outgoing messages however, the "mime" property set in the JSON and the data that is returned by this function may be different. In fact, very often the original JSON does not even hold any MIME data, and it is exactly the task of the plugin to generate the MIME based on other properties in the MIME. Exactly for that reason, this function normally returns NULL if you call it from a plugin that intercepts outgoing messages - unless some other plugin has already set the data using the MQ_setMime() function.
For more info, see the documentation about MQ_setMime.
MailerQ leverages RabbitMQ queues and exchanges to handle email processing and message flows. This setup provides flexibility in how emails are managed and delivered and how results can be routed and processed. Below, we outline how to configure this topology, with configuration options tailored to your needs.
Queue: A list where messages wait until they are processed by a consumer. MailerQ uses an "outbox" queue from which it reads the emails that need to be sent.
Exchange: A distribution point in RabbitMQ where messages are published. The exchange determines, based on the routing key, which queue(s) will receive a message. MailerQ normally sends all its messages to the same exchange.
Routing Key: A specific key that is used when messages are published to RabbitMQ. MailerQ uses different routing keys for different types of messages, such as retries, and success and failure reports.
Binding: A link that directs messages from an exchange to a queue or even another exchange. Bindings can have a routing key pattern, so only messages with matching routing keys are delivered to the connected queue or exchange. Bindings between exchanges allow for advanced message flows by chaining exchanges with different routing logic.
Using these concepts you can setup advanced topologies that allow you to route delivery reports to your own processing tools, and send retries and received messages back into MailerQ.
MailerQ reads its email instructions from a single, specified queue, commonly referred to as the "outbox" queue. All mails that need to be delivered should be placed in this queue. The following config file setting determines the name of this queue:
rabbitmq-queue: outboxMailerQ only consumes from this queue, and does not directly post new messages to it. When MailerQ wants to send or reschedule mail, it sends it to an exchange with a routing key. With the right topology, such messages then end up in the outbox queue.
In older versions of MailerQ this setting was named rabbitmq-outbox, and was used to specify both the queue, and the routing key for publishing.
All results from email processing, including retries and failures, are sent back to RabbitMQ. Also inbound mails, bounces, and messages that are received via other ways (directory pickup, API, etc), are sent to the exchange. In the config file the exchange can be set with this setting:
rabbitmq-exchange: mailerqBy default, the name of the exchange is not configured, and messages are published to the "default exchange", which is a special exchange in RabbitMQ with limited routing capabilities: it treats the routing key as a queue name, and places each message in the queue with the name of the routing key. If such a queue does not exist, the message is lost. When you leave the exchange empty, all routing keys in the config file are effectively queue names.
For advanced routing options, it is strongly recommended to change this setting and use a named exchange. We also recommend to only let MailerQ publish messages to this exchange, and create additional exchanges and bindings for your own publish operations. This will simplify monitoring and debugging, as you can be certain that all traffic through the exchange is guaranteed to come from MailerQ.
Routing keys are used for categorizing messages based on their purpose. MailerQ uses different routing keys for various operations, allowing messages to be routed to other tools or back into MailerQ. If you do not specify an exchange in the config file, the routing keys are effectively queue names.
The following config file settings can be used to set the routingkeys used for scheduling mails:
rabbitmq-outbox: outbox
rabbitmq-retries: retriesThe "outbox" routing key is used to put a message in the outbox. The "retries" routing key has the same purpose, with the difference that it is only used to schedule retries, for example when a 4xx SMTP response code was received or when a mail server was otherwise unreachable.
The recommended topology is to bind both routing keys to the outbox queue: all rescheduled and retried messages do then indeed end up in the outbox again. Additionally, you can bind the "retries" routing key to a separate queue of your own, which allows you to monitor temporary failures.
When a mail was delivered, or when it permanently failed, the result is published back to RabbitMQ with routing keys configurable by these settings:
rabbitmq-success:
rabbitmq-failure:If you leave the settings empty, which is the default, MailerQ will not publish delivery results. Do not bind the routing keys to the outbox queue, because the delivery results are only meaningful for further processing by external tools, and not to be picked up again by MailerQ.
Temporary failures are not published back with one of these routing keys, but are sent back to the outbox with the special "retries" routing key. To monitor these retries, set up an extra binding for this routing key into one of your own queues.
MailerQ can handle inbound SMTP connections. All messages that are received on its inbound SMTP port are published to RabbitMQ, using routingkeys set by these config file options:
rabbitmq-inbox: outbox
rabbitmq-local:
rabbitmq-reports:
rabbitmq-refused:The inbox routing key is for historical reasons standard set to "outbox", so inbound mail follows the same RabbitMQ routing as regular retried mail and is immediately scheduled for delivery. It is recommended to set this to a different value (like "inbox") and create an extra binding for this routing key to let the email flow back to the outbox. This has the same end result (inboud mail gets scheduled for delivery), but improves your monitoring capabilities.
The "local", "reports" and "refused" settings are optional. If you leave them empty, MailerQ simply publishes all incoming message with the inbox routing key. Once you do configure one or more of these extra routing keys, MailerQ starts analyzing and categorizing inbound messages, and publishes them with special routing keys based on this.
MailerQ internally keeps a list of "local" email addresses. This list can be edited via the management console. If you have configured a "local" routing key, and a mail comes in for an address on that list, the message is published with the "local" routing key.
If the local email also turns out to be not a normal type of email, but some kind of delivery report (or some other kind of report), the message is published with the "reports" routing key.
To summarize:
The last listed routing key is "refused". It is used for messages that were not accepted. For example, if you configure MailerQ to listen to one or more SMTP ports, and you require incoming connections to be authenticated, you might receive messages over unauthenticated connections. These messages are rejected and do not end up in RabbitMQ. However, for debugging and/or security reasons you might still want to find out who is flooding your SMTP server. By assigning a value to "rabbitmq-refused", you instruct MailerQ to publish rejected messages with a special routing key so that you can route them to a queue where you can inspect them.
In the above sections we described the queue from which MailerQ reads its outgoing messages (the outbox queue) and the exchange and routing key to which all incoming messages are published. However, besides handling incoming and outgoing messages, MailerQ can also construct and send email messages all by itself: Delivery Status Notifications (DSN).
A delivery status notification is an email message that is sent back to the original envelope address when a delivery fails (technically, it is also possible to send such notifications on successful delivery and/or when a message gets delayed, but in practice it is mostly used for failure notifications). By default MailerQ does not send such notifications because it uses result queues and JSON to report delivery results. However, if you add a "dsn" property to the JSON of an outgoing message, you can instruct MailerQ to send delivery status notifications too.
Technically, a delivery status notification is a regular email, and MailerQ simply posts a message to RabbitMQ when such an email has to be delivered. The routing key that is used can be controlled with this setting:
rabbitmq-dsn: outboxThe default setting is outbox, so it follows the same process as regular mails. If you want to intercept these notifications, or route them differently, you can install a different routing key.
MailerQ’s configuration file specifies queues and routing keys, and leaves it up to the user to set up this topology in RabbitMQ, with possible extra queues and bindings for handling results. This flexibility allows you to decide if and how to process messages routed by these keys, creating customized workflows that best fit your email processing needs. We encourage you to create your own topology, which gives you maximum control over the queues and bindings in RabbitMQ.
You have several options for setting up your topology:
Additionally, MailerQ can automatically create queues and bindings if desired.
MailerQ can infer the desired topology from the config file and, upon startup, create exchanges, queues, and bindings based on this configuration. Note that this process is based on assumptions, as the config file defines routing keys but not specific queues or bindings. To enable this feature, use the following settings in your configuration file:
rabbitmq-declare: true
rabbitmq-durable: true
rabbitmq-lazy: falseWhen rabbitmq-declare is set to true, MailerQ creates exchanges, queues and bindings at startup. You can control the persistence of these resources with rabbitmq-durable (to keep them after a RabbitMQ restart), and with rabbitmq-lazy you can opt to create lazy queues, which store messages on disk to ensure predictable performance under load.
When automatically setting up the topology, MailerQ follows these conventions:
The rabbitmq-results and rabbitmq-retry settings are retained for compatibility with earlier versions of MailerQ, where results and retries were directly queued rather than routed. These settings are no longer used when running MailerQ, but are still respected during initialization to create and bind queues.
Setting rabbitmq-lazy to true makes MailerQ declare its queues as lazy. All messages are then written directly to disk, ensuring performance stability at the cost of some speed. With lazy queues, RabbitMQ is consistently slow, which is sometimes preferrable over being slow during peak loads only. For more details, see the RabbitMQ documentation on lazy queues.
If the standard topology that is created by MailerQ is not sufficient, you need to create the topology yourself. Our amqpinit tool can be of help. It reads JSON input holding the toplogy setup. The following input JSON can for example be used for setting up a straight forward topology:
[{
"type": "exchange",
"name": "mailerq",
"exchangetype": "direct",
"durable": true
}, {
"type": "queue",
"name": "outbox",
"durable": true
}, {
"type": "queue",
"name": "failures",
"durable": true,
"arguments": {
"x-max-length": 10000
}
}, {
"type": "queue",
"name": "success",
"durable": true,
"arguments": {
"max-max-length": 10000
}
}, {
"type": "binding",
"exchange": "mailerq",
"queue": "outbox",
"routingkey": "outbox",
}, {
"type": "binding",
"exchange": "mailerq",
"queue": "outbox",
"routingkey": "retries"
}, {
"type": "binding",
"exchange": "mailerq",
"queue": "outbox",
"routingkey": "inbox"
}, {
"type": "binding",
"exchange": "mailerq",
"queue": "failures",
"routingkey": "failure"
}, {
"type": "binding",
"exchange": "mailerq",
"queue": "success",
"routingkey": "success"
}The above JSON takes care of the following:
If you use AMQP-init, or a different technology, for setting up the topology, remember to change your MailerQ config and set rabbitmq-declare to false, because you no longer need MailerQ to do this for you.
MailerQ picks up messages from the outbox queue, tries to send them out and then publishes the results back to result queues. Other programs or scripts can read out these result queues and further process the failures and succesful deliveries. The messages on the result queues are all JSON encoded.
MailerQ uses four different queues to which results are published. Based on the type of result (success, failure or retry) a message is published to one or more of these queues:
The "results" queue is used by MailerQ to publish all the results: both the successful results as well as the failures. The "success" and "failures" queues are used for just the successes and just the failures. These queues receive a subset of the messages published to the "results" queue. The "retry" queue is used for transient results for deliveries that have not yet failed or succeeded, and that are going to be retried.
Depending on your needs, you can enable or disable these different queues (for example, if the only thing you're interested in are the failures, you do not configure the results, success and retry queues, and only set the failures queue). The result queues can be configured in the RabbitmQ section in the config file.
A typical delivery scenario is this: you first publish your JSON message to the outbox queue. MailerQ picks up this message and tries to send it. If this send attempt fails (because the receiving server rejected it with a soft error), MailerQ modifies the initial JSON to include info about this first soft error, and publishes it back to the outbox queue to retry the delivery later. The message is also sent to the retry queue to notify external programs that the message will be retried.
After a while, MailerQ picks up the message from the outbox queue again for a second attempt. This time the delivery succeeds and the message is modified once more to also contain info about this correct delivery, and the message is published to both the success queue and the generic results queue.
The messages that MailerQ publishes to the results queues are JSON encoded, and hold the same properties as the original message that was read from the outbox queue, including all the custom properties that you added (thus: if you added some custom identifier to the initial JSON object, it will also be present in the JSON object that ends up in the result queue.
The "mime" property is often stripped from the initial JSON before it is published to a result queue, so that these queues do not take up too much space from RabbitMQ. However, if you want to keep the mime data in the JSON, you can add a "keepmime" property. For more info about the supported properties, check the JSON specification for outgoing messages.
Besides stripping the MIME data, MailerQ also adds a "results" property to the JSON. This "results" property holds an array with information about each delivery attempt that was taken. A result message will thus look something like this:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"results": [
{
"state": "intro",
"result": "error",
"time": "2016-02-04 13:45:15",
"mta": "mta1.example.com",
"from": "1.2.3.4",
"to": "10.11.12.13",
"messages": 1,
"sent" : {
"connection": 355,
"data": 0
},
"code": 451,
"status": "5.0.0",
"description": "Please come back later",
},
{
"state": "message",
"result": "accepted",
"time": "2016-02-04 14:01:25",
"mta": "mta2.example.com",
"from": "1.2.3.4",
"to": "10.11.12.14",
"messages": 4,
"sent" : {
"connection": 5876,
"data": 4923
},
"code": 250,
"status": "2.0.0",
"dsn": true,
"description": "Requested mail action okay, completed",
}
]
}The "results" property holds an array of JSON objects. Each object contains the result of one delivery attempt. Every result object can have the following properties:
| state | state in the delivery where the error occured |
| result | type or result (accepted,timeout,lost,error,invalid) |
| time | time of the result |
| mta | name of the receiving MTA (reported when the connection was set up) |
| from | ip from which the mail was sent |
| to | ip to which the mail was sent |
| authentication | if authentication/login was necessary for access, the authentication protocol used |
| cipher | the encryption protocol used (if the message was sent over a secure connection) |
| certificate | information about the ssl certificate that was issued by the receiving server |
| messages | number of messages sent over the same connection |
| sent/connection | Amount of bytes sent over the same connection |
| sent/data | Amount of bytes of MIME (no SMTP commands) sent over the same connection |
| code | SMTP result code |
| status | extended SMTP status code |
| description | answer received from receiving server |
| dsn | did the remote server implement the dsn protocol? |
Note that some properties are optional. For example, the properties "to", "from", "mta" and "messages" are only used if an actual TCP connection was set up and are not present if the mail failed because of a DNS lookup failure. The "code", "status" and "description" properties are only used when the remote server returned an error code and/or human readable description to explain why the message was or was not accepted.
The "mta" property holds the name of the MTA that accepted the message. To get this MTA name, MailerQ reads the answer from the initial welcome message that the receiving server sent right after the connection was set up.
The "dsn" property is only used for succesful deliveries and tells you whether the incoming server supports the DSN extension. If this is set to true, and you had also included a DSN setting in your input JSON to request a success notification, you can expect a notification from the server that accepted the message.
For logging purposes, MailerQ also extracts some data from the MIME when publishing the emails back. This way, these properties can be logged using the flexible log format.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"results": [
{
"state": "message",
"result": "accepted",
"time": "2016-02-04 14:01:25",
"mta": "mta2.example.com",
"from": "1.2.3.4",
"to": "10.11.12.14",
"messages": 4,
"code": 250,
"status": "2.0.0",
"dsn": true,
"description": "Requested mail action okay, completed",
}
],
"extracted": {
"from": "my-sender-address@my-domain.com",
"to": ["info@example.org", "other@example.org"],
"subject": "Test Subject",
"headers": {
"x-example-property": "some-mime-property"
},
"envelope": {
"domain": "bounce.my-domain.com",
"local": "my-sender-address"
},
"recipient": {
"domain": "example.org",
"local": "info"
}
}
}As is shown in the example, some properties are exposed in a parsed way (from, to, subject), as well as unparsed exposure of all the MIME-headers that were present in the (personalized) email.
MailerQ logs the state in which the error occurs (this is the "state" property), and the type of result in that state. An email that is processed by MailerQ goes through the following states:
| process | the input is checked and assigned to the from and to ip address between which it will be sent |
| suppression | Suppression List is checked |
| storage | the mime data is loaded from external nosql storage |
| responsive | the json message is converted into a responsive email |
| personalize | the message is personalized if personalization data is available |
| rewrite | the email is being rewritten |
| dns | the hostname to which the mail should be sent is looked up in DNS |
| bind | a local IP address is chosen from which the mail is going to be sent |
| connect | a tcp connection is set up |
| handshake | a handshake is being performed with the proxy server |
| intro | the TCP connection has been set up, waiting for initial welcome message from server |
| ehlo | the "EHLO" message has been sent |
| helo | the "HELO" message has been sent |
| starttls | the "STARTTLS" message has been sent |
| authplain | the "AUTH PLAIN" message has been sent |
| authlogin | the "AUTH LOGIN" message has been sent |
| authusername | the username for authentication has been sent |
| authpassword | the password for authentication has been sent |
| authcram | the "AUTH CRAM-MD5" message has been sent |
| authresponse | the response to the cram-md5 challenge has been sent |
| mailfrom | the "MAIL FROM" message has been sent |
| rcptto | the "RCPT TO" message has been sent |
| data | the "DATA" message has been sent |
| message | the full mime data followed by a dot has been sent |
| idle | smtp pipe was not processing any messages |
| unknown | the connection is in an unknown state, for example when a pipeline failed |
During all of the above states errors might occur. If this happens, the state of the delivery and the type of error is set added to the result JSON. A successful delivery has gone through all the states, and is published to the result queue with the property "state" set to "message" and "result" set to "accepted".
During all the above states errors can occur. The type of error is logged in the "result" property. It always has one of the following values:
| accepted | message was accepted (only reported in combination with state "message") |
| timeout | no answer was received in time |
| lost | connection was lost while waiting for an answer |
| invalid | an answer was received, but it could not be recognized as valid answer |
| error | an answer was received, but the answer contained a fatal error message |
The SMTP protocol requires communication between two mail servers: a sending server and a receiving server. MailerQ is the sending server, and sends instructions to the receiving server (like "MAIL FROM", "RCPT TO", et cetera). After each of these instructions it waits for an answer from the remote server.
If the answer is not received in time, the "timeout" result is used. It is also possible that the receiver did sent back an answer, but that the answer could not be recognized as a valid SMTP response message. In such a case we use the "invalid" result type. When the TCP connection was lost while waiting for an answer the "lost" type is result, and the "error" type is used in case the receiver server sent back a valid SMTP answer, but the answer was that the message was rejected.
If MailerQ's instruction was accepted, the delivery moves to the next state, and no result is logged in the JSON object.
Not every result state is linked to a state in the SMTP protocol. The following combinations can also occur, and should be interpreted in a special way:
| state | result | |
| process | timeout | incoming message was expired (maxdelivertime or maxattempts exceeded) |
| process | error | the input json contains a mail that could not be assigned to an ip address (for example because no sufficient secure ip exists) |
| process | invalid | the input json contains a mail that could not be sent (for example due to license constraints, or because the recipient is missing or not parsable) |
| storage | lost | smtp connection was lost while loading data |
| storage | invalid | smtp response was received while loading data |
| storage | error | no data was found |
| responsive | error | responsiveemail.com algorithm could not turn json into mime |
| dns | error | dns lookup failure (domain does not exist / no ip addresses available for accepting mail) |
| bind | error | no local ip addresses available for sending mail |
MailerQ does its best to fill in an appropriate error code in the result object too. Be aware that the error code and status code sometimes come from the remote server and is blindly copied by MailerQ to the result JSON, while in other situations it is generated by MailerQ.
If the receiving server supported STARTTL, and MailerQ managed to secure the connection, the result object also holds an "cipher" and "certificate" key. The "cipher" holds the settings of the encryption algorithm that was used, while the "certificate" holds the settings of the SSL certificate that was issued by the receiving server. Both the "cipher" as the "certificate" are stored as JSON objects, holding all sorts of sub values.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "john.doe@gmail.com",
"results": [
{
"state": "message",
"result": "accepted",
"time": "2016-07-20 14:01:25",
"mta": "mx.gmail.com",
"from": "1.2.3.4",
"to": "10.11.12.14",
"messages": 1,
"code": 250,
"status": "2.0.0",
"dsn": false,
"description": "Requested mail action okay, completed",
"cipher": {
"name": "ECDHE-RSA-AES128-GCM-SHA256",
"bits": 128,
"algbits": 128,
"version": "TLSv1\/SSLv3",
"id": 49199
},
"certificate": {
"notbefore": "2016-07-13 14:40:53",
"notafter": "2016-10-05 14:17:00",
"version": 3,
"verified": true,
"result": "ok",
"keyusage": 1,
"self-signed": false,
"subject": {
"common": "mx.google.com",
"country": "US",
"locality": "Mountain View",
"state": "California",
"organization": "Google Inc"
},
"issuer": {
"common": "Google Internet Authority G2",
"country": "US",
"organization": "Google Inc"
},
"altnames": [
"gmr-smtp-in.l.google.com",
"gmr-mx.google.com",
"gmail-smtp-in.l.google.com",
"aspmx5.googlemail.com",
"aspmx4.googlemail.com",
"aspmx3.googlemail.com",
"aspmx2.googlemail.com",
"aspmx.l.google.com",
"alt4.gmr-smtp-in.l.google.com",
"alt4.gmail-smtp-in.l.google.com",
"alt4.aspmx.l.google.com",
"alt3.gmr-smtp-in.l.google.com",
"alt3.gmail-smtp-in.l.google.com",
"alt3.aspmx.l.google.com",
"alt2.gmr-smtp-in.l.google.com",
"alt2.gmail-smtp-in.l.google.com",
"alt2.aspmx.l.google.com",
"alt1.gmr-smtp-in.l.google.com",
"alt1.gmail-smtp-in.l.google.com",
"alt1.aspmx.l.google.com",
"mx.google.com"
]
},
}
]
}With the Response Pattern system, properties can be added to the result json. This is useful for your own classification and processing of messages, but there are also some special properties that can be used to override error handling logic.
| fatal | should be the message be retried, or was this result already a permanent failure |
| individual | override whether or not an error applies to a single message, or rather to any message sent to the domain |
| strange | is this is a strange type of result |
| greylisted | should this result be sent from the same ip later |
| fallback | should this result trigger a fallback server |
These properties are not set by MailerQ itself, but can be set by setting these properties in a Response Pattern. If they are not present, MailerQ tries to figure out these properties by reasoning about the response it got from the server.
If you use Swift Mailer as your main mail engine in your project you can download MailerQ Transport Plugin to extend functionality of the Swift Mailer. Below you can find an example showing how to combine those two projects together.
example.php<?php
// load required dependencies
require_once 'swiftmailer/lib/swift_required.php';
require_once 'MailerQPlugin/MailerQConnection.php';
require_once 'MailerQPlugin/MailerQTransport.php';
// an array with additional options
$options = array(
'ips' => ['1.2.3.4','1.2.3.5'],
'maxattempts' => 5
);
// set up a connection to AMQP server
$connection = new MailerQConnection('localhost', 'outbox', 'guest', 'guest', '/');
// create the message using Swift_Message
$message = Swift_Message::newInstance();
$message->setSubject('This email was sent using Swift MailerQ Transport Plugin');
$message->setBody('Example message');
$message->setFrom('example@domain.com', 'Example');
// deliver the message to
$message->setTo('example@domain.com');
// create a new MailerQTransport instance
$transport = new MailerQTransport($connection, $options);
// create a new SwiftMailer instance
$mailer = Swift_Mailer::newInstance($transport);
// send the actual message using the MailerQTransporter
$mailer->send($message);
?>
This example assumes that MailerQ is running and configured. It uses the .NET/C# AMQP client library provided by RabbitMQ.
The .NET/C# example consists of two files; one that takes care of placing messages on the outbox queue and one that takes sent messages from the result queue.
SendThis class connects to RabbitMQ and places a JSON encoded message on the outbox
message queue. To connect to RabbitMQ and to create the JSON encoded message,
the program uses the private variables defined inside the class. If you want to
use the .NET/C# example to test your MailerQ configuration, you will have to run
the Send class before running the Result class.
Send.cs// declare the namespaces
using System;
using RabbitMQ.Client;
class Send {
/**
* Settings for RabbitMQ
*/
// same as the rabbitmq-host option in the config file
private static string hostname = "localhost";
// same as the rabbitmq-user option in the config file
private static string username = "guest";
// same as the rabbitmq-password option in the config file
private static string password = "guest";
// same as the rabbitmq-vhost option in the config file
private static string vhost = "/";
// same as the rabbitmq-outbox option in the config file
private static string outbox = "outbox_test";
/**
* Settings for sending out the test email
*/
// domain where the test message should be delivered
private static string recipientDomain = "example.org";
// email where the test message should be delivered
private static string recipientEmail = "testmail@example.org";
// address where the email was sent from
private static string fromAddress = "this_is_my_email@my_domain.org";
/**
* Main method
*/
public static void Main()
{
// create the connection factory
ConnectionFactory factory = new ConnectionFactory();
// set the hostname of RabbitMQ
factory.HostName = hostname;
// set the user name to connect to RabbitMQ
factory.UserName = username;
// set the password to connect to RabbitMQ
factory.Password = password;
// set the virtual host to RabbitMQ
factory.VirtualHost = vhost;
// create the connection
using (IConnection connection = factory.CreateConnection())
// create the channel
using (IModel channel = connection.CreateModel()) {
// declare the queue
channel.QueueDeclare(outbox, true, false, false, null);
// put the domain inside the JSONMessage
string message = "{ "\"envelope\": \"" + fromAddress + "\", "
+ "\"recipient\": \"" + recipientEmail + "\", "
+ "\"mime\": \"" + "From: " + fromAddress + "\r\nTo: "
+ recipientEmail
+ "\r\nSubject: Example subject\r\n\r\n"
+ "This is the example message text\" }";
// get the byte array value of the message
byte[] mime = System.Text.Encoding.UTF8.GetBytes(message);
// publish the message on the outbox queue
channel.BasicPublish("exchange1", "key1", null, mime);
// send some output to the console
Console.WriteLine("Sent", message);
}
}
}ResultThis class connects to the RabbitMQ server and gets the messages that were
placed on the outbox message queue by the Send class back from the result
message queue. The result messages from the result queue are shown to the user.
The Result class can only output any relevant information, if it is executed
after the Send class was run. The Result program will keep running, waiting for
new messages until it is terminated (use Ctrl-C). You could run the Send program
from an other terminal, to test the effect.
Result.cs// declare the namespaces
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
class Result {
/**
* Settings for RabbitMQ
*/
// same as the rabbitmq-host option in the config file
private static string hostname = "localhost";
// same as the rabbitmq-user option in the config file
private static string username = "username";
// same as the rabbitmq-password option in the config file
private static string password = "password";
.NET/C#
// same as the rabbitmq-vhost option in the config file
private static string vhost = "/";
// same as the rabbitmq-outbox option in the config file
private static string resultbox = "results_test";
public static void Main() {
// create the connection factory
ConnectionFactory factory = new ConnectionFactory();
// set the hostname of RabbitMQ
factory.HostName = hostname;
// set the user name to connect to RabbitMQ
factory.UserName = username;
// set the password to connect to RabbitMQ
factory.Password = password;
// set the virtual host to RabbitMQ
factory.VirtualHost = vhost;
// create the connection
using (IConnection connection = factory.CreateConnection())
// create the channel
using (IModel channel = connection.CreateModel()) {
// declare the queue
channel.QueueDeclare(resultbox, true, false, false, null);
// start the consumer
QueueingBasicConsumer consumer = new QueueingBasicConsumer(channel);
// start the consumer
channel.BasicConsume(resultbox, true, consumer);
// message telling the user that the script is waiting for messages
// that will be placed on the result queue by MailerQ
System.Console.WriteLine("Waiting for messages." + "To exit press CTRL+C");
// keep waiting for new messages
while(true) {
// get a delivery from the result queue
BasicDeliverEventArgs ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
// get the message from the result message queue as a
// byte array
byte[] data = ea.Body;
// convert the message into a string
string message = System.Text.Encoding.UTF8.GetString(data);
// write some output to the console
System.Console.WriteLine("Received:");
System.Console.WriteLine(message);
System.Console.WriteLine("------------------------------------------");
}
}
}
}Provided .NET/C# is working and you installed the .NET/C# AMPQ library in the
/lib/ directory, you can test your configuration as follows
$ gmcs -r:/lib/bin/RabbitMQ.Client.dll Send.cs
$ gmcs -r:/lib/bin/RabbitMQ.Client.dll Result.cs
Run the Send.exe file to put a message on the queue.
$ MONO_PATH=/lib/bin mono Send.exe
Run the Result.exe file to get the result messages from the result message queue.
$ MONO_PATH=/lib/bin mono Result.exe
Function that can be implemented by a plugin and that is called by MailerQ every time a MIME message is received over a SMTP connection
bool mq_smtp_in_data(MQ_Context *context, MQ_Connection *connection, const char *mime, size_t size);
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
MailerQ can be configured using the "/etc/mailerq/config.txt" config file. Most of the variables and settings in this config file are discussed in separate articles. However, the following settings are not mentioned anywhere else:
max-delivertime: 86400
max-attempts: 0
retry-interval: 600,600,1800,1800,3600,3600,7200
server-id: 1
license: /path/to/license.txt
user: username
lock: /path/to/lock.txtMost of the settings speak for themselves. A small overview.
When a mail gets greylisted, or when there is an other type of soft failure, MailerQ reschedules the mail for later delivery. Inside the JSON data of each message you can add the optional properties "maxdelivertime" and "maxattempts" that instruct MailerQ how long it should retry to deliver the message, and when it should give up trying.
If you do not include these properties, MailerQ falls back to the "max-delivertime" and "max-attempts" config file settings. The "max-delivertime" variable holds the number of seconds MailerQ should keep on trying after the message was consumed for the first time. The "max-attempts" setting holds the total number of attempts before giving up.
You should normally set the "max-attempts" value to 0, meaning unlimited. MailerQ will then only give up after the number of seconds set in the "max-delivertime" variable, no matter how many attempts are made.
If a mail is published back to the outbox queue to be tried again, MailerQ uses the "retries" property in the JSON to find out how long it should wait for the next attempt. Because this setting is JSON based, it is possible for each message to have its own retry interval. This could be useful if you send both transactional as well as commercial mass mailings. For transactional emails you normally want small intervals because these mails should be delivered as soon as possible.
If the "retries" property is not set in the input JSON, MailerQ falls back to the setting from the config file. The config file variable "retry-interval" is used for that. This setting should hold a comma separated list of the delays between the attempts. This should be set to a number of seconds.
The default value is "600,600,1800,1800,3600,3600,7200". This means that the second and third attempt is sent 10 minutes after the previous attempt. The next two attemps use an interval of half an hour, then an interval of one hour for the next two attempts, and all subsequent attempts are sent after three hour intervals. You can change variable to anything you want.
By default, any injection method causes MailerQ to strictly check whether or not an email address is actually fully RFC compliant, and an error will occur if this is not the case. On SMTP injection, the message will simply be rejected, but on AMQP injection, the message will not be tried.
This can be loosened by the following properties.
validate-envelope: loose (default: strict)
validate-recipient: loose (default: strict)Setting these to loose causes the address to be interpreted as loosely as possible,
only requiring the @ sign in the middle and a valid domain on the right. The local-part
may be anything, and will not be restricted by the RFC any more. Note that although
attempts will be made, the other server might reject the message anyway, since it might
be in violation with the standard.
Each message that is accepted by MailerQ gets a unique message ID. To prevent that two running MailerQ instances both assign exactly the same ID to a message, you can assign a "server-id" variable. This identifier guarantees that message ID's will never conflict.
server-id: 1To work properly, MailerQ needs a license file. The license file can be downloaded from the MailerQ website. You can store the file anywhere on the file system. The path to the license file can be configured by setting
license: <Path to your license> (empty by default)If you have configured MailerQ to use ports lower than 1024 (like port 25 for SMTP and/or port 80 for the management console), the MTA must be started as user "root". Once the ports have been opened, MailerQ changes its identity to the user set in the config file.
user: <user name> (empty by default)The username to change identity to after the SMTP and HTTP ports have been opened.
To set a proxy for outgoing HTTP requests, the "proxy-server" variable can be used.
proxy-server: protocol://hostname:port (empty by default)Currently, two proxy types are supported, HTTP and SOCKS. For example, to route all HTTP
requests through an http proxy, the required uri would be http://example.org:8888, for a
proxy running on the host example.org on port 8888. For the SOCKS proxy, the URI would be
socks://example.org:8888 for a SOCKS proxy running on port 8888.
For the HTTP proxy, the proxy itself should support the CONNECT request. The HTTP proxy will be used as a tunnel, so this should be the case for both secure and insecure requests. The DNS gets resolved by MailerQ, so the final outgoing request will be similar to the request below.
CONNECT 1.2.3.4:80 HTTP/1.1To prevent that MailerQ starts more than once, MailerQ stores its process ID (pid) in a lockfile. The name and location of the lockfile can be set in the configuration file.
lock: <filename> (default: /tmp/mailerq.pid)MailerQ sends a message to Copernica hourly. This message contains some usage statistics
for that hour, e.g. the number of errors that occured. This information is used
to improve MailerQ. Users with a paid license can disable this message by adding
the heartbeat property to the config file:
heartbeat: false (default: true)This structure is provided for plugin functions that work with message contents. It can be used to both retrieve and update message properties. If you use it for updating messages, you should be aware of the phase in the SMTP protocol that your plugin is intercepting: it is for example pointless to change the envelope address if the "MAIL FROM" instruction has already been sent.
All members in this MQ_Message structure are private, to interact with this structure, accessor functions have been created.
A MQ_Message pointer is passed to your plugin when a message is received on the incoming SMTP port, and when it is about to be sent over an outgoing SMTP connection. If your plugin implements one of the following functions, it can get access to a message, retrieve information from it, or make modifications to it.
| Function name | Description |
|---|---|
| mq_smtp_in_message() | Called by MailerQ when a message is received on an incoming SMTP connection |
| mq_smtp_out_data() | Called by MailerQ right after it has sent the "DATA" command and is about the send MIME data |
The MQ_Message structure is essentially a wrapper around the original JSON data that was loaded from RabbitMQ (if your plugin interacts with outgoing messages), or a wrapper around the JSON data that is going to be published to RabbitmQ (for incoming messages). Your plugin can get access to the JSON data, and make modifications to it.
| Function name | Description |
|---|---|
| MQ_json() | Get access to the JSON data of a message |
Besides the JSON, you can also access derived properties. You do not have to inspect the JSON directly, but can use one or more of the following methods to get access to the message properties:
| Function name | Description |
|---|---|
| MQ_envelope() | Get access to the envelope address |
| MQ_setEnvelope() | Set the envelope address |
| MQ_recipient() | Get access to the recipient address |
| MQ_setRecipient() | Set the recipient address |
| MQ_mime() | Get access to the MIME data |
| MQ_setMime() | Get the MIME data |
MailerQ has a non-blocking nature, and uses callbacks to pass control to a plugin. It is then up to the plugin to interact with the event loop, and pass control back to MailerQ when it is ready with its task. Passing back control can be done in three different ways:
The MQ_retry() implements the third behavior, and is probably the most peculiar way of handing back control to MailerQ. If you call it, MailerQ will start all over, and will call all plugins again.
This is a very odd way of handing back control, and you normally should not use it, because you run the risk of ending up in an infinite loop. It could however be useful if your plugin makes a change to a received message, and you want to run all plugins again so that they can all respond to the modified message.
/**
* Finish plugin processing
*
* @param connection the connection that may move on to the next plugin
*/
void MQ_retry(MQ_Connection *connection);See also MQ_continue() and MQ_complete() for the other (and probably more useful) ways of handing back control.
Internally, MailerQ works with libev for registering timers and watching file descriptors and while some simplified wrapper functions are available for the MQ_Context, you might find it easier to work directly with libev instead.
/**
* Retrieve the underlying libev event loop from
* the given context
*
* @param context the context to retrieve the loop from
* @return struct libev
*/
struct ev_loop *MQ_ev(MQ_Context *context);
The loop instance that is returned will already be running and since it is likely to be handling many other smtp and database connections it is highly discouraged to stop this loop, since it will most likely break other open connections. You should only use it for adding timers and filedescriptors to it.
MailerQ runs on Linux. To get MailerQ up and running on your own computer or on a server you need to take a couple of simple steps:
You can test if everything is setup correclty by using the Hello World example
MailerQ comes with a web based management console that you can use to monitor exactly what is happening, and to adjust all your delivery settings.
If you want to change the envelope address that is associated with a MQ_Message, you can use this function. This function allows you to store a new envelope address in a message.
/**
* Update the envelope address
*
* @param message the message to set the envelope address in
* @param envelope actual envelope address
* @param size length of the envelope string
*/
void MQ_setEnvelope(MQ_Message *message, const char *envelope, size_t size);
This function has a different behavior if you call it on a message _that is being received_ and on a message _that is being sent_. If you call it on a message that is being received, for example inside your [mq_smtp_in_message()](mq_smtp_in_message) function, it is simply an alias for setting the "envelope" property in the JSON. Changing the envelope address by calling this function, will also change the JSON that is going to be published to RabbitMQ.
For outgoing messages however, changing the envelope with this function will not modify the JSON data. If your plugin calls this function before the "MAIL FROM" instruction is sent, the newly set envelope address is going to be used in the SMTP protocol, but when the message is published back to RabbitMQ (for example to the result queue, or back to the outbox queue for a retry) it will still have the original envelope address in the JSON. If you also want to modify the envelope address in the JSON data, you should use the [MQ_json()](mq_json) function as well.The MQ_Connection structure is passed to all callbacks dealing with SMTP connections. It allows your plugin to find out information about the connection, and to send data over the connection.
There is a whole list of functions that your plugin can export that deal with connections.
| mq_smtp_in_connect() | Called after an incoming TCP connection is established. This is the first time that the MQ_Connection is passed to the plugin, and a good place for per-connection initialization code. |
| mq_smtp_in_secure() | Called when the STARTTLS instruction is received and the connection is turned into a secure connection |
| mq_smtp_in_authenticate() | Called when the SMTP client sends the login+password credentials. Allows you to run your own authentication checks. |
| mq_smtp_in_mailfrom() | Called when the "MAIL FROM" command is received |
| mq_smtp_in_rcptto() | Called when the "RCPT TO" command is received |
| mq_smtp_in_data() | Called when the "DATA" command is received |
| mq_smtp_in_message() | Called when a message is received. When multiple "RCPT TO" addresses were given, this function is called once for each address. |
| mq_smtp_in_close() | Called after an incoming SMTP connection is closed. This is the last time that the MQ_Connection is passed to the plugin. You can use this function for cleanup code. |
The above functions are only called for MQ_Connection pointers that represent incoming connections.
All members in the MQ_Connection structure are private. To interact with the structure the following accessor functions have been created.
| Callable function | Description |
|---|---|
| MQ_fileDescriptor() | Retrieve the filedescriptor linked to a connection |
| MQ_send() | Send data over a SMTP connection |
| MQ_authenticated() | Returns whether this is an authenticated connection |
| MQ_setAuthenticated() | Mark a connection as an authenticated connection |
| MQ_secure() | Returns whether this is a secure TLS connection |
| MQ_connectionData() | Retrieve the data that is associated with the SMTP connection |
| MQ_setConnectionData() | Associated data with the SMTP connection |
If you run multiple MailerQ instances, you can set up a MailerQ cluster. All instances will be aware of the other running MailerQ servers, share information and hand over undeliverable messages. Setting up a cluster has many advantages:
Every MailerQ server in the cluster shares its configuration with the other instances. This allows the management consoles to link to each other, and more importantly, to trigger other instances to reload cached data when settings change. For example, when you add a DKIM key via the management console of your first MailerQ server, the other MailerQ servers will also have direct access to this new DKIM key.
Another advantage of setting up a cluster is that email messages automatically end up in the right outbox queue. In a normal cluster setup, you run multiple servers and on each server you run a single MailerQ instance. Each instance reads from its own outbox queue (instead of having all instances consume from the same queue, which is not recommended). If you want to send out an email from a specific IP address, you therefore must make sure that the message ends up in the right outbox queue. However, if you do publish a message to a wrong queue, the MailerQ inter-cluster communication ensures that the message will be moved to the right outbox queue.
There are two config file settings relevant for setting up a cluster:
cluster-address: amqp://login:passwd@host/vhost
cluster-exchange: clusterBoth cluster settings are optional. If you leave them empty, the exchange "cluster" is used, and the same server address as is set in the "rabbitmq-address" setting.
It is important that every running MailerQ instance uses the same cluster settings. Even when your instances all have their own private RabbitMQ server, the instances must still share the same values for the "cluster-address" and "cluster-exchange" settings.
It is also recommended to use the same relational database for each instance. By doing this, the instances use the same delivery throttles and DKIM Keys.
If your RabbitMQ server supports secure connections, you can configure MailerQ to connect to an "amqps://" address instead. The communication between MailerQ and RabbitMQ will then be encrypted.
cluster-address: amqps://guest:guest@hostname/vhost
cluster-verify: falseMailerQ checks the server certificate when it connects to a secured RabbitMQ server. If this certificate can not be verified with one of the known openssl certificate authorities, MailerQ refuses to set up a connection. If you use a self-signed certificate, you may want to skip this extra test, and set the config file option "cluster-verify" to "false".
All instances create a private message queue in RabbitMQ and bind this private queue to the cluster exchange. By doing this, each MailerQ instance receives all messages that are published by the other instances. Each instance also periodically sends its local IP addresses and some other configuration settings to this exchange. The announcement messages end up in all the private queues of the instances, and everyone in the cluster receives information about everyone.
When a change is made via one of the management consoles, a message is sent to the cluster exchange, so that each server in the cluster can reload the delivery limits from the database.
Be aware that both RabbitMQ and MailerQ support clustering, but that from a technical standpoint this clustering has a whole different meaning. Clustering for RabbitMQ means that queues are mirrored across different RabbitMQ instances, so that no messages are lost when one of the RabbitMQ servers crashes.
For MailerQ clustering means something different: the individual servers communicate and share data with each other, but it does not necessarily mean that emails will still be delivered when one of the servers crash. If no other server has access to the same IP addresses as the crashed server, there is no way that any other server can take over the deliveries. If you do want to achieve this, you have to set up a heartbeat daemon so that other servers can automatically jump in and take over when one of the servers fails.
To find out if the cluster is working, you can simply go to the management console of one of your MailerQ instances. If the console shows hyperlinks to your other MailerQ instances, you know that the instances have identified each other over the shared RabbitMQ exchange. You can also start MailerQ with the "--list-cluster" command line argument to find out if the cluster is working:
$ mailerq --list-clusterIf you start MailerQ like this, it will connect to the cluster and wait for one second. All the MailerQ instances that announce themselves within that second are displayed. Normally, all instances in the cluster report within this timeout. However, if your servers or internal network is overloaded, the 1 second period could be too limited. In that case you can change the "cluster-timeout" variable to use a different timeout. This setting can be set in the config file or via the command line:
$ mailerq --list-cluster --cluster-timeout=5You can also write your own scripts or programs to access the cluster. You can for example write an application that connects to RabbitMQ, and that publishes a single JSON message to the cluster exchange to instruct all running MailerQ instances that they should reload the database:
{ "reload": true; }If the above instruction is received by a MailerQ instance, it will immediately connect to the database and reload all settings from it.
It's also possible to intercept all cluster traffic. For this you simply have to create a queue in RabbitMQ and bind it to the shared cluster exchange. All messages that are sent to the cluster will then also end up in your queue. By reading out the messages from the queue, you will exactly see what is going on.
Tip: normally all MailerQ instances report once every couple of minutes their status on the cluster. If you do not want to wait that long, you can send a single "hello" json message:
{ "hello": true; }For the Hello World Example to run we first need to make sure we can use the Management Console of MailerQ and the SMTP port is reachable. First we start up our installed version of MailerQ
To start RabbitMQ as an administrator, start the server for RPM-based systems using:
sudo systemctl enable rabbitmq-server
sudo systemctl start rabbitmq-serverMailerQ uses port 80 to show the Management Console. This is the default HTTP port. Any Apache or nginx servers will most likely use this port. You can stop them or you can adjust the HTTP port that MailerQ uses. Just modify the www-port variable in MailerQ config.
# Management console configuration
www-port: 8485MailerQ also runs a SMTP server and it will try to use port 25 for it. That means that you have to stop any other SMTP servers before starting MailerQ (e.g. postfix). Alternatively, you can change MailerQ's SMTP port. Just modify the smtp-port variable in MailerQ config (e.g. 2525 will work just fine).
smtp-port: 2525Within this config file you can configure MailerQ, to change the Management Console password for example. Although all other config file settings have decent defaults, you might want to take a look at them.
After you connected MailerQ to RabbitMQ you can start MailerQ
sudo systemctl start mailerqGo to your localhost or localhost:8485 to access the MailerQ Management Console and send your first email with MailerQ!
The simplest example to send a message using MailerQ is via the cli:
echo -e "from: your_email@domain.com\nto: your_email@domain.com\nsubject: MailerQ example message\n\nHello World" | mailerq --extract-recipients --ignore-dotIf you are fast enough, you can track this message in the management console.
Mailerq-amqpinit is a simple Linux application for setting up queues and exchanges in RabbitMQ. It is an application independent from MailerQ, and can be helpful in, for example, startup scripts to ensure that a RabbitMQ server is correctly configured. It takes a JSON string from standard input, and creates all the exchanges and queues that are defined in that file.
The application is stored in our APT repository. If you have already enabled our APT repository, you can install it with this command:
sudo apt install mailerq-amqpinitYou typically run this script with one command line argument, and you pipe the JSON input into the program:
amqpinit amqp://user:password@hostname/vhost < input.jsonThe amqp:// address holds the address of a RabbitMQ server.
The JSON input file typically holds a JSON array with all the resources (queues, exchanges and bindings) that are to be created. For example:
[ {
"type": "queue",
"name": "my-queue",
"durable": true,
"arguments": {
"x-queue-type": "classic"
}
}, {
"type": "exchange",
"name": "my-exchange",
"durable": true,
"arguments": {
"x-max-length": 1000
}
}, {
"type": "exchange",
"name": "my-other-exchange",
"exchangetype": "fanout",
"durable": true
}, {
"type": "binding",
"exchange": "my-exchange",
"queue": "my-queue",
"routingkey": "my-routing-key"
}, {
"type": "binding",
"exchange": "my-exchange",
"target": "my-other-exchange",
"routingkey": "my-routing-key"
} ]For each item in the array, the following properties are supported:
Not all properties are meaningful for all resource types. For example, the "exchangetype", "queue", "target" and "routingkey" properties only make sense for exchanges. The properties "durable", "autodelete", "passive", "exclusive" and "internal" are all booleans and are available for completeness because the AMQP protocol supports them. Realistically, only the "durable" property is useful.
MailerQ-Mimify is an application to preprocess messages before they are picked up by MailerQ. The JSON messages that are normally read by MailerQ can first be processed by the MailerQ-Mimify application to create rich formatted emails based om properties inside the JSON.
Normally, MailerQ expects email messages to be formatted as MIME strings when they are injected into RabbitMQ. This means that before you inject a message into RabbitMQ/MailerQ, you must ensure that it is RFC2822 compliant. If you send the messages however first to the mimify application, you can leave the MIME generation up to MailerQ-Mimify, and focus just on the message content rather than the format.
For example, if you send the following JSON message through MailerQ-Mimify, it will be picked up and converted to full MIME:
{
"envelope": "johndoe@example.com",
"recipient": "info@example.com",
"mime": {
"subject": "This is my subject line",
"from": {
"name": "Joe Doe",
"address": "johndoe@exampel.com"
},
"to": {
"name": "Example Company",
"address": "info@example.com"
},
"content": "<html><body>Hello World!</body></html>",
"text": "Hello World!"
}When MailerQ-Mimify reads out the above message from RabbitMQ, it converts it into a new JSON document and publishes it back to RabbitMQ in a MailerQ compatible format. The above document is for example converted into this:
{
"envelope": "johndoe@example.com",
"recipient": "info@example.com",
"mime": "MIME-Version: 1.0\r\nSubject: This is my subject line\r\nFrom:\r\n...."
}For readability, we have shortened the full MIME string in the above example.
The MailerQ-Mimify application supports many features:
The MailerQ-Mimify application was introduced in the beginning of 2024 to turn the MailerQ MTA into a more microservice oriented application. Up to version 5.14.*, MailerQ itself was capable of doing JSON-to-MIME conversion. However, as the number of mimify features grew, it was decided that the responsibility of generating MIME strings could better be handled by a standalone application.
The application is stored in our APT repository. If you have already enabled our APT repository, you can install it with this command:
sudo apt install mailerq-mimifyAfter that, you can run the application with the command mailerq-mimify. The configuration is normally picked up from its config file, but can also be given through command line arguments.
All options can be supplied in the system wide config file, via environment variables, and as command line options. The config file is stored in /etc/copernica/mimify.txt. The following options are all equivalent:
All other configuration settings can be converted in a similar manner.
The mimify tool reads all instructions from a RabbitMQ message queue, and writes the results back to RabbitMQ. The following options can be used to configure this:
Note that both the input and output queues can also be in "exchange:routingkey" format.
In a normal setup, you set the output address to MailerQ's inbox exchange and queue. This ensures that messages generated by MailerQ-Mimify are immediately delivered by MailerQ. The input address can be set to whatever queue where your own software stack publishes its messages to.
Inside the JSON you can refer to resources that are downloaded from external locations, like images or content feeds. To finetune the downloader, the following options are supported:
The blacklist and whitelist options can be used to restrict the IP addresses from which downloads are supported. If you keep the whitelist empty, downloads from all IP addresses are permitted, except for the blacklisted IP's. With a whitelist, downloads can only come from one of the listed IP's.
The proxy address can be set to a URL of a proxy server. The URL can be in the format "http://hostname:port" or "socks://hostname:port".
The mimify application runs a small webserver that exposes metrics in Prometheus format. It can be configured with the following config file settings:
The webserver only serves one page, a Prometheus compatible metrics page.
The variables used for personalization can be altered with modifiers. A modifier
is called on the variable by appending it with a | before the closing curly
brace. E.g. if you want to apply the tolower modifier on
variable {$name} you use: {$name|tolower}.
An example of how the modifiers can be used for a personalized mail is:
Hello {$name|escape},
Your name {$name|escape} is {$name|strlen} characters long.
Bye!Besides calling one modifier on a variable it is possible to chain modifiers.
E.g. you have name data. However, the strings containing names, are sometimes
capitalized and sometimes not. You want to use these names and you want to
be sure that a name starts with a capital and the rest of the name is in
lower case. Doing this is easy. You use {$name|tolower|ucfirst}. The first
modifier tolower, will make sure that the name is all in lower case. Then
modifier ucfirst is applied, which will capitalize the first character.
Note that if you call a modifier that does not exist, no effect takes place.
The following table lists all supported modifiers:
| Modifier | description |
|---|---|
| base64_encode | base64 encoder |
| base64_decode | base64 decoder |
| cat:"string" | concatenates a string to variable |
| count | count number of elements in variable |
| count_characters | count number of characters in a string |
| count_paragraphs | count number of paragraphs in a text (by counting newlines) |
| count_words | count number of words in a text |
| default:default value | use default value if variable is not set |
| empty | check whether a variable is empty |
| escape:"string" | escape html characters (or other chars) inside a string |
| indent:num = 1:char = " " | put num whitespaces in front of every line |
| md5 | perform md5 hashing |
| nl2br | replace newlines with html br tags |
| range:start = 0:end | truncate list to get the items between positions start and end |
| regex_replace:regex:replace_text | replace substrings using regular expression |
| replace:"string1":"string2" | replace occurrences of string1 with string2 |
| sha1 | perform sha1 hashing |
| sha256 | perform sha256 hashing |
| sha512 | sha512 hashing |
| spacify:separator = " " | place a separator between every input character |
| strlen | count the characters in a string |
| strstr:"substring":before = false | return the string starting from the first occurrence of substring if before = false. otherwise return the string until the first occurrence. |
| substr:start position:length | return the substring from start position onward, optionally truncated after length characters |
| tolower | convert all characters to lower case |
| toupper | convert all characters to upper case |
| trim | trim the white space and endline characters off both sides of the input |
| truncate:length = 80:etc = "...":break_words = false | truncate inputs that are longer than length and append etc at the end. break_words = true allows truncating parts of words |
| ucfirst | replace first character with an upper case character |
| urlencode | encode input for use in an url |
| urldecode | decode input for use in an url |
With this modifier you encode the data into base64. Note that this modifier does not have an effect on arrays. Usage:
The base64 encoding of {$name} is {$name|base64_encode}.With this modifier you can decode base64 encoded information. Usage:
The decoded information is {$base64encoded|base64_decode}With this modefier you can concatenate a string to your variable. If the variable is an arrary, the string will be used. Usage:
{$name|cat:"string"}With this modifier you can count the number of elements in an array. If the variable is not an array a 0 will be returned. Usage:
{$names|count}With this modifier you can count the number of characters in your text. If this modifier is called on an variable that contains an array, 0 is returned. Usage:
{$name|count_characters}With this modifier you can count the number of paragraphs in your text. If this modifier is applied on an array a 0 is returned. Usage:
The following text has {$text|count_paragraphs} paragraph.
Text:
{$text}With this modifier you can count the number of words in your text. If this modifier is applied on an array a 0 is returned. Usage:
"{$text}" has {$text|count_words} words.With this modifier you can set a default value that will be used if the value does not exist. Usage:
This will always show {$name|default:"something"}With this modifier you can check if the variable is set or not. It will return true if the variable is set or false if it isn't. Usage:
{if $name|empty}
Dear customer,
{else}
Dear {$name},
{/if}With this modifier you can escape, or encode, the variable. The form is: escape:"value". The possible values are: "html", "url", and "base64". If no value is specified "html" is used. On an array this modifier is ingored. Usage:
{$text|escape:"html"}
is equal to:
{$text|escape}With this modifier you can add indentation to your text. You can specify the amount of indentation and the character(s) that is used for indentation. The syntax is indent:num:char, where "num" is the number of characters and "char" the character. The default of these is 1 and space respectively. On an array this modifier is ignored. Usage
{$text|indent:4:" "}With this modifier you get the MD5 checksum of your text. If you use it on an array, the MD checksum is calculated over the entire array excluding the keys. Usage:
{$text|md5}This modifier replaces your newlines with the equivalent HTML br tags. This enables you to write plain text that will be truncated in html mode. On an array this value is ignored. Usage:
{$text|nl2br}With this modifier you can take a range form your input if your input is an array. The form is range:start:end, where start is the start position and end is the exclusive end position. If the variable is not an array, the value will be ignored. Usage:
{$array|range:2:5}With this modifier you can replace parts of your text, base on a regular expression, with other text. If the variable is an array this value will be ignored. Usage
This will replace each number in the variable string with the string " a number "
{$text|regex_replace:"\d":" a number "}With modifier replace you can replace a part of your text with some other text. The syntax is: replace:"string1":"string2", where all occurrences of "string1" will be replaced by "string2". When the variable is an array, this modifier does not have an effect. Usage:
{$text|replace:"hi":"hello"}With this modifier you get the SHA1 hash of your text. If the variable is an array the SHA1 hash will be calculated over the entire array, excluding the keys. Usage:
{$text|sha1}With this modifier you get the SHA256 hash of your text. If the variable is an array the SHA256 hash will be calculated over the entire array, excluding the keys. Usage:
{$text|sha256}With this modifier you get the SHA512 hash of your text. If the variable is an array the SHA512 hash will be calculated over the entire array, excluding the keys. Usage:
{$text|sha512}With this modifier you can add a character, or characters, between each character in your variable. The syntax is spacify:separator, where the default separator is space. If the variable is an array the modifier will be ignored. Usage:
{$text|spacify:"."}With this modifier the length of the text in a variable can be obtained. If the variable is an array a 0 will be returned. Usage:
{$text|strlen}With this modifier you can search for a string in you variable and get the first occurrence plus the rest of your variable, or you can get everything in the variable till the searched string. The syntax is: strstr:string:before, where string is the searched string, and before is a boolean, default = false. If the boolean is true, you get what is in the variable before the first occurrence of string. If it is false you get what is in the variable from the first occurrence of string onwards. If the variable is an array the modifier is ignored. Usage:
If the variable holds "Hello world!", this will print Hello, {$variable|strstr:" ":true}
and this will print world!, {$variable|strstr:"w":false}, just like this
{$variable|strstr:"w"}With this modifier you can obtain a substring from you variable. The syntax is substr:start:length, where start is the starting position in the string (zero indexed) and length is the length you want to obtain. The default for length is the the length of the string minus the starting position. Usage:
If the variable is "0123456789" this will print 2 to 9
{$variable|substr:2}
and this will print 456
{$variable|substr:4:3}With this modifier you can change all character in your text to lowercase characters. Usage:
{$text|tolower}With this modifier you can change all characters in your text to uppercase characters. Usage:
The next part looks like it is shouted {$text|toupper}With this modifier you can trim trailing white space and new line characters from your text. The characters that are trimmed are: spaces, tabs, newlines, carriage returns, vertical tabs, and end of strings. Usage:
{$text}With this modifier it is possible to truncate your text to a certain length. If the text is truncated it is possible to add some characters to indicate that the text is truncated. These extra characters are counted for the truncation length as well. You can also specify whether it is allowed to break words or not. The syntax is: truncate:length:etc:break_words, where length has a default of 80, etc (the replacement), has a default of ... and break_words has as default false.
Usage:
{$text|truncate:50:"....":true}With this modifier you replace the first character of your text with an uppercase character. Usage:
{$name|ucfirst}With this modifier you can url encode your text. Usage:
{$text|urlencode}With this modifier you can url decode your text. Usage:
{$text|urldecode}Some mail servers initially seem to accept a message, but send back a bounce email later on in which they inform you that the mail could not be delivered after all. Such asynchronous bounces are sent back to the envelope address of the initial message. There is sadly often no way to find out whether a server is going to send back notifications, and what these messages are going to look like.
However, there is a formal standard for these bounce messages: the Delivery Status Notification (DSN) format. And there is an extension to the SMTP protocol that allows mail servers to exchange with each other whether they respect this standardized format, if the original sender would like to receive such bounce messages, and what kind of information he or she wants to receive. This standard make it much easier to link incoming bounces to the original sent email.
MailerQ supports this standard and SMTP extension: when communicating with a receiving server, DSN parameters can be passed to tell the receiver what kind of bounces should be returned. If MailerQ is configured to receive mail on a SMTP port, it understands these kind of parameters too. MailerQ can also be configured to send out, receive and recognize DSN messages.
Delivery Status Notifications are sent back to the envelope address of your messages. This is the address that is used in the "MAIL FROM" command of the SMTP handshake, and that is stored in the "envelope" property of the input JSON. If you want to receive DSN messages, you therefore first have to make sure that your envelope address is valid, and that it indeed points back to your server.
Besides the envelope address that is used to send back DSN messages, some mail servers (but not all!) have implemented the SMTP DSN extension, and allow extra parameters to be passed to instruct them what kind of DSN messages you would like to receive back. You can for example specify that you want to see the full original mail message in the notification, or just the headers. You can also specify whether you want to receive notifications on failure, on success and/or on a delayed delivery.
{
"envelope": "youraddress@yourdomain.com",
"recipient": "info@example.com",
"mime": "....",
"dsn": {
"notify": "FAILURE",
"ret": "HDRS",
"envid": "unique-identifier",
"orcpt": "info@example.com"
}
}The above JSON sets "youraddress@yourdomain.com" to be the envelope address. If something goes wrong, the notification will be sent to this address. If the receiving mail server also supports the DSN SMTP extension, MailerQ uses the "dsn" property in the input JSON to pass additional DSN parameters to the receiver. You can use the following four DSN settings:
| notify | comma separated events that trigger a notification (FAILURE, DELAY, SUCCESS, NEVER) |
| ret | should the notification hold the full original mail or just the headers (FULL, HDRS) |
| orcpt | the address to be included in the notification as "original-recipient" |
| envid | unique identifier to be included in the notification as "original-envelope-id" |
The "notify" variable specifies what sort of events trigger the Delivery Status Notification. Possible values are "NEVER", "SUCCESS", "FAILURE" and "DELAY". If you set it to "FAILURE", you will only receive bounces on failure. It is also allowed to set this to a comma separated list of values. If you set the "notify" property to "SUCCESS, FAILURE", a bounce message will be sent on successful delivery as well as on failure.
The bounce message holds a full delivery report, as well as the full original message. If you want to save a lot of network bandwidth, you can set the "ret" property to the value "HDRS". By doing so, you instruct MTAs not to include the full MIME message in the status notification, but only the original MIME headers.
The property "orcpt" is an optional property and holds the original recipient address. This original recipient address is included in the delivery report. This property is optional; if you leave it out the actual recipient will be used. This property is only useful if you want the DSN to contain a different "original-recipient" value than the actual recipient.
The last property that you can include in the "dsn" object, is "envid". This is a message specific message identifier, and you can set it to whatever you like. It will be included in the bounce message as "original-envelope-id", and you can use it to match the bounces with a sent message.
Keep in mind that not every mail server supports the DSN extension, and that even if they do support it, they may not always respect your parameters. There is no guarantee that you receive the DSN that you wanted. For example, if you specify that you want to receive the full original mail in the bounce (you set "ret" to "FULL"), it still is possible that you do not receive a bounce at all, or a bounce with just the headers.
In the above example, we've demonstrated the JSON properties for passing DSN parameters. If you inject mails using a non-json format (via SMTP, the spool directory or via MailerQ's command line interface), you can use MIME headers instead. The following MIME headers are recognized, and do the same thing as the JSON settings:
x-mq-dsn-notify: FAILURE
x-mq-dsn-ret: HDRS
x-mq-dsn-envid: your_unique_identifier
x-mq-dsn-orcpt: email@example.comIf one or more of the DSN settings are missing, the default values from the global config file are used. These defaults can be set with the following config file variables:
dsn-notify: FAILURE
dsn-ret: HDRSYou can only set defaults for the "notify" and "ret" settings, because the "envid" and "orcpt" are normally only used on a per-message setting and it is not meaningful to have default values that apply to all emails.
MailerQ has its own ways to report the results: it publishes JSON result objects to the result queues, and it writes results to log files. These reporting methods are powerful, and easy to integrate. Most of the MailerQ users therefore do no let MailerQ send out its own DSN messages back to the envelope address. The only DSN messages that are normally sent back to your envelope address are notifications from MTAs that initially accepted the mail.
However, MailerQ can be configured to send out DSN messages too. If you set up a RabbitMQ DSN queue (using the "rabbitmq-dsn" config file property), MailerQ starts sending out DSN messages. Every time that a message cannot be delivered, an outgoing bounce mail is constructed and published to this DSN queue. If you assign the outbox to this "rabbitmq-dsn" setting, the bounces are even automatically picked up and delivered.
We wrote that a bounce is sent on every failed delivery. This is technically not completely correct. To be precise, a bounce is created everytime that the message specific "dsn.notify" setting matches. Thus, if "notify" is set to "SUCCESS", MailerQ sends out a bounce message on successful delivery, and when it is set to "FAILURE", bounces are sent on failure. The "notify" setting is normally set to "FAILURE" so in normal circumstances bounces are sent on failure.
If you set the "rabbitmq-dsn" setting to an empty string (which is the default), no Delivery Status Notifications are sent by MailerQ. But MailerQ does forward the DSN settings to the receiving MTA, so you could still receive DSNs from MTAs further up the chain.
If you inject mail into MailerQ using SMTP, you can pass DSN parameters with the "MAIL FROM" and "RCPT TO" commands.
MAIL FROM:<envelope@example.com> RET=HDRS ENVID=unique_identifier
RCPT TO:<info@example.com> NOTIFY=FAILURE ORCPT=rfc822;info@example.comThese parameters are directly copied to the "dsn" property in the JSON that is published to the inbox queue. To enable or disable this feature, use the "dsn-advertise" setting in the config file.
dsn-advertise: 1If you set dsn-advertise to 1, MailerQ announces in its SMTP connection handshake that it supports the DSN extension, and converts the DSN parameters that are passed to the "MAIL FROM" and "RCPT TO" commands to JSON properties. If you set it to 0, no DSN parameters can be passed to "MAIL FROM" and "RCPT TO".
MailerQ can also be used to recognize incoming DSN messages. If you submit a DSN message to the SMTP port of MailerQ, and MailerQ detects that both the recipient address is on the list of Local Email Addresses and that the message holds a report message, it publishes this report to the queue for incoming reports. The name for this queue can be set in the config file with the "rabbitmq-reports" variable.
This means that you can not only use MailerQ to send out mass mailings, but that you can also run a MailerQ instance to process the delivery reports that come back. If you use "reports@yourdomain.com" as envelope address, you can run a MailerQ instance to process all mails for the yourdomain.com domain. Via the management console you add the "reports@yourdomain.com" local email address, so that all these incoming messages are accepted and sent to either the "local" message queue for normal email messages, and the "reports" message queue for emails that were recognized as reports.
MailerQ does not only recognize DSN messages, but also other types of reports like DMARC reports and disposition notifications.
MailerQ relies on the DNS resolver functions that are standard installed on your system. The bad thing about these standard functions is that they are not very good, and they were never intended for high performance MTAs that do many DNS lookups at the same time. There is however also a nice thing about using the standard resolver functions: the DNS lookups of MailerQ return exactly the same results as the lookups done by system utilities like telnet, ping, openssl, dig, et cetera.
Since version 5.11, MailerQ relies on DNS-CPP internally. This is an asynchronous resolver that can handle many different concurrent requests efficiently, and also has support for DNSSEC. This library picks up /etc/resolv.conf and /etc/hosts normally, similar to getaddrinfo().
Unfortunately, for some domains the TTLs that are in DNS are very low, causing MailerQ to retry much too often. Some domains specify TTLs in the range of 0-30 seconds. To mitigate this, the minimum respected TTL is 60 seconds by default, since some domains specify 0 TTL which can cause lookups for every message. This incurs a significant performance penalty. To change this minimum value, the following config file option can be used:
dns-min-ttl: 120This will make sure that the mininum respected TTL is 120 seconds. Note that the minimum allowed value should be greater than zero, since zero is explicitly disallowed due to the performance penalty.
Since 5.12, the timeouts on DNS requests can be configured in MailerQ. There are two settings for altering the timings. Naturally, settings from /etc/resolv.conf will be picked up first, and then the settings in MailerQ are applied on top of them. If the setting inside MailerQ is empty, the setting from resolv.conf will be kept.
dns-timeout: 60.0
dns-interval: 3.0The dns-timeout setting limits the time the resolver waits for a response after the last attempt was made.
The dns-interval specifies after how many seconds of waiting for a reply on the initial query, the query should be retransmitted.
Since 5.13 three settings were added to finetune the maximum workload of the DNS resolver:
dns-attempts: If you run into errors for domains that are reported to not exist, while you know that they do exist, consider making this value higher. MailerQ will then try longer doing DNS lookups before giving up. Default: 5.dns-capacity: The number of DNS lookups that can run in parallel. Make this value higher if you notice that DNS lookups are a bottleneck in your delivery, and make it lower if your DNS server cannot keep up with the lookups (or add more DNS servers). Default: 100.dns-sockets: To improve security MailerQ rotates the lookups over different sockets. By doing this, MailerQ avoids that the same port is constantly used for handling DNS responses. If this is set to 1, MailerQ can only rotate / start using a different port when there are no more lookups in progress. A higher value brings more rotating options and thus more security. Default: 4.DNS works over UDP, and so these packets may be lost at any point in the process, which means that they may have never been
seen by the recursor, or the recursor has replied and we had to drop the packet. To prevent the latter problem, the dns-buffersize
setting was introduced.
dns-buffersize: 1MBThis sets the receiving and sending buffer size. We recommend raising this as high as possible, since that makes the probability of dropped packets because the receive buffer is full significantly lower. Keep in mind that this is bound by the system limit on the receive buffer size, which can be altered with sysctl. To make it larger, use the following commands to tune the settings.
sysctl -w net.core.rmem_max=1000000
sysctl -w net.core.wmem_max=1000000If all nameservers in /etc/resolv.conf are local resolvers and have a similar resolving speed, we recommend setting the rotate option to true:
dns-rotate: trueThis will allow MailerQ to distribute the load across the nameservers randomly, performing a rudimentary load-balancing on it.
In addition to the MailerQ application, we have some tools and scripts that (may) come in handy for MailerQ users - and for others. The following tools are available:
The amqpinit and httpcaller tools are generic: they can also be useful in a non-MailerQ environment.
Application to preprocess messages before they are picked up by MailerQ. This application converts JSON messages into RFC8822 complient MIME messages. This application is helpful if you want to specify your messages in JSON format instead of MIME format. You can put properties like the subject line, headers, actual content and attachments into the MIME message specification, which are then turned into a valid MIME messages that is then sent to MailerQ to be delivered.
Click here for more information about AmqpInit
This is a generic tool to create RabbitMQ queues and exchanges based on a JSON config file. It reads the input from the config file, and ensures that all the listed queues, exchanges and bindings are created. It can be useful in startup or initialization scripts to ensure that all your queues are correctly configured. Also useful as standalone application.
Click here for more information about AmqpInit
This is a generic tool for sending out HTTP POST calls. Based on JSON input from a RabbitMQ message queue, this application sends out HTTP POST calls. This is a generic standalone tool for many different use cases - not only in combination with MailerQ. The JSON messages in the queue should contain the URL to call, and the message body to post. It can also optionally hold extra headers and other properties for the outgoing HTTP calls.
Click here for more information about HttpCaller
This is a MailerQ specific middleware application that connects the results-queue from MailerQ with the input-queue of the MailerQ-httpcaller (see above). It converts MailerQ's result queue output into instructions that are compatible with the httpcaller. For example, it adds a URL property to the JSON, and the message body is formatted. This "amqphttp" tool takes care this:
Click here for more information about AmqpHttp
A MailerQ specific setup script to simplify setting up a MailerQ webhook stack. Although you can configure a stack yourself using the above three tools, you can also leave it up to the mailerq-webhook script:
Incoming messages are published to either the local message queue, or to the regular inbox queue. In a normal setup, this is normally used to decide whether the email is going to be delivered to a local address, or that it is going to be relayed.
If you write a plugin that intercepts incoming messages, you can update the "local" property by calling this method. If you set it to true, the message is going to be processed as a local email and will be published to the local message queue.
/**
* Update the local setting
*
* @param message the message to change
* @param local new setting
*/
void MQ_setLocal(MQ_Message *message, bool local);This function is only meaningful for incoming messages. You can call it on outgoing message too, but that will not change the behavior of MailerQ. This function is therefore only used in plugins that implement the mq_smtp_in_message() function.
MailerQ depends on RabbitMQ for message queueing. Before you can even start MailerQ, you first need a running RabbitMQ instance. The www.rabbitmq.com website has all the information you need for setting up RabbitMQ. We do however have some tips, tricks and recommendations for setting up RabbitMQ with MailerQ.
The RabbitMQ version that is installed in the repository of your operating system might be outdated. You really need a version that is up-to-date, because MailerQ uses a couple of RabbitMQ features that were only recently added. We therefore recommend downloading and installing RabbitMQ directly from the www.rabbitmq.com website instead of using the version that comes with your OS.
Click here to download and install RabbitMQ.
The RabbitMQ installation has to be at least version 3.3.1+ for MailerQ to be able to connect to it.
By default, RabbitMQ is installed with a user with login guest and password guest.
These are the login credentials that you have to include in the configuration file
of MailerQ. However, this guest/guest login only works for clients that connect
to RabbitMQ locally (from the same machine). If you run MailerQ and RabbitMQ on
different servers, the guest/guest login does not work. Therefore, if you install
RabbitMQ and MailerQ on different machines, you either need to add a user with a
different name and password, or you should configure RabbitMQ to allow guest/guest
logins from remote hosts as well.
To allow remote guest/guest logins, you can use the loopback_users setting in the
RabbitMQ config file. By including this option in the RabbitMQ config file, you tell
RabbitMQ that it is ok for clients to login with guest/guest, even if the connection
comes from a remote location. If you do include this setting, please make sure that
you also have a firewall running, because you do not want everyone from all over the
internet to connect to your RabbitMQ instance!
Read more about setting up loopback_users.
If you're using RabbitMQ 3.8.15 or higher you need to disable the consumer_timeout setting. This is a hardcoded global setting that has been enabled in newer RabbitMQ versions, and requires all messages to be acknowledged within 30 minutes. However, this does not work well for MailerQ, as deliveries are sometimes paused or retried for (much) longer than this 30 minute limit.
To disable the timeout you have to create an advanced.config and add the following lines:
[
{rabbit, [
{consumer_timeout, undefined}
]}
].The location of the advanced.config file depends on your operating system. More information about the advanced configuration file can be found here.
Read more about the Delivery Acknowledge Timeout
Just like MailerQ, RabbitMQ comes with a very nice web interface. However, this web interface is not enabled by default, and must be explicitly configured. We recommend doing this, because it is much easier to control RabbitMQ via a web browser, than with command line tools.
Read more about setting up the browser interface..
MailerQ not only uses RabbitMQ to fetch the messages that it is going to send, but also publish back the delivery results (if you have this configured). If you do not process these delivery results in time, the queues in RabbitMQ will get fuller and fuller and you run the risk that your RabbitMQ server runs out of resources (memory or disk space), which might crash the server. This can especially happen in a production environment, where many messages are published and consumed.
So, when you run MailerQ in production, do make sure that you have set up cronjobs or other scripts that periodically or continuously process the messages from the result queues.
Send raw data directly to the SMTP connection.
MailerQ takes care of its own output buffering, so you can be sure that all data is always delivered, unless the connection is suddenly closed by the peer.
/**
* Send data over a TCP connection
* @param connection the connection in which the data was stored
* @param buffer data to size
* @param size size of the buffer
* @return size_t number of bytes sent
*/
size_t MQ_send(MQ_Connection *connection, const void *buffer, size_t size);
Function to associate plugin data with a SMTP connection. This data void-pointer is completely ignored by MailerQ, but can be useful for plugins that want to preserve data between calls to the plugin callback functions.
The data pointer is segmented per plugin, so this function will not not overwrite data stored by other plugins.
If the data is allocated by yourself, you will also have to make sure that the data gets deallocated, for example by also implementing the mq_smtp_in_close() function, in which you deallocate the data after the connection is closed.
/**
* Store userdata in the connection
*
* @param connection the connection in which to store the data
* @param void* the data to store
*/
void MQ_setConnectionData(MQ_Connection *connection, void *data);If your plugin contains this function, it will be called as soon as the SMTP connection enters the data state, giving you the chance to manipulate the message body before it is sent.
Normally, the full MIME messages that are going to be sent by MailerQ are stored in the JSON objects that are loaded from RabbitMQ. When a SMTP connection has been set up, the MIME is extracted from the JSON and sent over the SMTP connection.
But you may write a plugin that does this in a different way. If you want to write a plugin to generate the MIME in a different fashion, you can do so by implementing the mq_smtp_out_data() function. This function is called by MailerQ after the SMTP connection has been set up, to load the actual MIME message.
This method is designed to run asynchronously. You should not execute any blocking calls from within this function, if you do you will disturb other active connections resulting in timeouts.
#include <mailerq.h>
/**
* Load a MIME message
*
* @param context Access to the context
* @param connection Access to the SMTP connection
* @param message Access to the JSON message
* @return boolean Do we accept control from here?
*/
MQ_EXPORT bool mq_smtp_out_data(MQ_Context *context, MQ_Connection *connection, MQ_Message *message) {
// in most cases, plugins implementing this function will alter the
// message contents before it is sent. for this, you can use the
// MQ_setMime() function. other message properties can of
// course also be updated, but at this stage in the sending process
// it most likely won't matter (we have already connected to the
// SMTP server and have given it the email address to which we want
// to send a message, so this cannot be changed).
// if we return true here, MailerQ will stop processing this message
// until either the MQ_complete, MQ_continue or MQ_retry function is
// called. if none of the functions is called, MailerQ will wait for the
// remote server to close the connection and mark it as an error.
return true;
}
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
MailerQ-Webhook is a utility script to simplify setting up the technology stack consisting of HttpCaller and AmqpHttp. When you run the webhook script, these applications will be automatically configured and started, and all relevant queues are automatically created in RabbitMQ.
The application is stored in our APT repository. If you have already enabled our APT repository, you can install it with this command:
sudo apt install mailerq-webhookOur webhook package is designed to provide a simple way to get started. After you install the webhook package you simply run 'mailerq-webhook', which will prompt you for the url to send the hooks to. It will then read out your mailerq configuration to setup both amqphttp and httpcaller and initialize all resources it needs.
mailerq-webhookThe script assumes you have already installed mailerq, and that the full MailerQ configuration is stored in /etc/mailerq/config.txt.
You can also do everything by hand. Please check out the documentation for the individual tools:
After your plugin is loaded, MailerQ starts making calls to it. To do this, MailerQ simply tries to locate specific functions in the plugin shared object file, and if it finds one or more of these functions, it will make calls to them every time a certain event occurs. There is one function that is required, mq_version(), but all others are optional. You only have to implement the optional functions if you want your plugin to override the default MailerQ behavior.
| Function name | Description |
|---|---|
| mq_version() | Returns API version number to ensure that MailerQ and plugin are compatible |
| mq_initialize() | Called right after the plugin is activated, allows plugin to run initialization code |
| mq_cleanup() | Called right before MailerQ closes, allows plugin to run cleanup code |
Before we continue, a single paragraph about the function naming convention of the MailerQ API. There are two type of functions: The first type of functions are the functions that your plugin can implement, and that are called by MailerQ when certain events occur -- these are functions like the ones listed in the table above. For these type of functions we use all lowercase characters. The other type of functions are functions that are offered by the MailerQ API, and that can be called by your plugin. These functions all have the uppercase "MQ_" prefix, and use CamelCase notation. The declarations of this second group of functions can be found in the "mailerq.h" header file.
MailerQ is a multi-threading application, with multiple worker threads that are all busy sending and receiving emails. If you write a plugin, it is important to realize this, because you cannot always safely access the same global data from your plugin functions. Locking might be necessary if you access global data from these multiple threads, and it could be useful to use per-thread data.
For every worker thread, MailerQ creates a MQ_Context structure that is passed to most of the plugin callback functions. For every worker thread that is created, and for every worker thread that is stopped, MailerQ calls the mq_context_initialize() and mq_context_cleanup() functions (but of course only if these functions exist in your plugin). You can implement these functions to run your own per-thread initialization or cleanup code.
| Function name | Description |
|---|---|
| mq_context_initialize() | Called after a worker thread is started, allows plugin to run thread specific initialization code |
| mq_context_cleanup() | Called right before a worker thread is stopped, allows plugin to run thread specific cleanup code |
If you want your plugin to interact with incoming SMTP traffic, you can implement one or more of the functions from the following table. When MailerQ loads your plugin, it tries to locate any of the following functions, and when it can find one or more of these functions, it will make calls to them every time the relevant event occurs. All functions are optional, you only have to write the ones that you need.
| Function name | Description |
|---|---|
| mq_smtp_in_connect() | Called after an incoming TCP connection is established |
| mq_smtp_in_secure() | Called after an incoming TCP connection is turned into a secure connection |
| mq_smtp_in_authenticate() | Called after someone sends login credentials over an incoming SMTP connection |
| mq_smtp_in_mailfrom() | Called after the MAIL FROM instruction was received |
| mq_smtp_in_rcptto() | Called after the RCPT TO instruction was received |
| mq_smtp_in_data() | Called after MIME data was received over the incoming connection |
| mq_smtp_in_message() | Called after the entire message was received and is turned into a JSON message |
| mq_smtp_in_close() | Called after an incoming SMTP connection is closed |
If you want your plugin to interact with outgoing SMTP traffic, you can add one or more of the functions from the following table to your plugin. MailerQ will call them every time a message is being sent.
| Function name | Description |
|---|---|
| mq_smtp_out_data() | Called by MailerQ when the connection enters data state |
| mq_smtp_out_sent() | Called by MailerQ after all mime data has been sent |
To reduce the load on RabbitMQ, MailerQ can use an external message store. In this case, only email metadata (the recipient, the envelope address, et cetera) has to be stored in the JSON object in RabbitMQ, while the full MIME data can be stored in the message store.
storage-address: mongodb://hostname/database/collection
storage-threads: 1
storage-policy: all
storage-ttl: 3600This message store is completely optional: if you set the "storage-address" variable to an empty string, MailerQ works just as well (even faster because no extra communication with the storage server is necessary), but the load on RabbitMQ and the network will be much higher.
When configured, MailerQ interacts with the message store in three situations:
MailerQ supports multiple storage engines: MongoDB, Couchbase, SQLite, MySQL and Postgres are the supported databases, where MongoDB is the preferred platform. Although it is possible to use MySQL, SQLite or PostgreSQL, we do not recommend using such relational databases, as those systems are not optimized for document storage. If you want to store the messages on disk, you can use a dir:// based backend. Messages can also be stored in AWS S3 buckets, which can be useful when running MailerQ on AWS. If you set the storage address to a "http" or "https" URL, the bodies will be stored and retrieved via HTTP requests.
The address of the message store can be set with the "storage-address" config file variable. The following addresses are supported:
storage-address: mongodb://hostname/database/collection
storage-address: couchbase://password@hostname/bucketname
storage-address: sqlite:///path/to/database/file
storage-address: mysql://user:password@hostname/databasename
storage-address: postgresql://user:password@hostname/databasename
storage-address: dir:///path/to/directory
storage-address: s3://accesskey:secretkey@region/bucketname
storage-address: http://hostname/path/to/dir
storage-address: https://hostname/path/to/dirIf you have a cluster of Couchbase or MongoDB servers, you can split the
hostnames with semicolons (;). For MongoDB you can also specify
the name of the replica set in the address if you have more than one server.
storage-address: couchbase://password@host1;host2;host3/bucketname
storage-address: mongodb://host1;host2;host3/replicaset/database/collectionMailerQ relies on third party libraries to be present on your system to communicate with the storage platform. The Couchbase C client library has to be installed if you want to use the Couchbase storage engine, and the Mongo C Driver is needed to connect to MongoDB. For the Sqlite, Mysql and PostgreSQL engines MailerQ also relies on the availability of the associated drivers on the system.
MongoDB is the recommended storage platform. The address string to connect to MongoDB is directly passed to the MongoDB driver. All the options that are supported by this driver can be used in the address string.
Click here for the MongoDB documentation
Sometimes MongoDB cannot succesfully retrieve data, even though it is available. In this case, it is useful to repeat the fetch operation a couple of times. To enable this feature, you can add a special option to the address:
storage-address: mongodb://hostname/database/collection?readAttempts=3The default number of attempts is 1. If you want to repeat failed lookups a couple of times, you can pass in a higher value.
To prevent that many small read operations are fired at MongoDB, MailerQ normally groups fetch operations into a single "multi-get" operation that fetches many documents in one go. This reduces the number of queries that are sent to MongoDB, but if one of the requested documents is hard to find, it also slows down the lookup of all the other documents in the same query. To get a balance between this, you can limit the number of fetch operations that are grouped together:
storage-address: mongodb://hostname/database/collection?maxQuerySize=10MongoDB has a limitation of around 16 MB per document (there is some overhead due to the usage of their internal BSON representation). If MailerQ has to store a bigger document, the message is split up into smaller parts that are all individually stored into MongoDB. As a consequence, you can find three types of documents in the database.
Most mails are less than 16mb big. So you will mostly see regular messages in the database with the following properties:
If the message is bigger than 16mb, MailerQ splits up the message and stores a master document with the following properties:
Each individual part of a big message has the following properties:
If you use the "directory://" storage backend, MailerQ stores all the messages in separate files on the file system. To prevent that the number of files in a directory becomes too big, MailerQ creates a nested directory structure. By default, this directory structure is four files deep. If you want to use a different depth, you can specify this via a "depth" parameter:
storage-address: directory:///path/to/directory?depth=3If you set your storage backend to a "http://" or "https://" address, the message bodies are retrieved via HTTP requests. This allows you to set up a REST service that loads the bodies from storage engines that are not natively supported by MailerQ. Or you can build a REST service that generates message bodies on the fly. Your REST service has to support the following methods:
If, for example, you set your storage address to "http://example.com/path", MailerQ will load all documents from URL's like "http://example.com/path/KEY" where "KEY" corresponds to the identifier of the message body.
The REST service must return full message body strings in MIME format. It is possible to compress them with GZIP compression. If your REST service sends out such compressed messages, make sure that you also add a "content-encoding: gzip" HTTP header. This same header is sent by MailerQ when it uploads message bodies.
Strangely enough, some storage drivers (MongoDB, MySQL, Sqlite and PostgreSQL) only offer synchronous, blocking drivers. The storage operations can only be executed sequentially, after previous operations were completed. To prevent that the entire MailerQ process gets blocked while a storage operation is in progress, MailerQ opens multiple connections to the storage servers, and starts separate threads in which the operations are being executed.
The number of threads (and the number of storage connections) can be set using the "storage-threads" variable. If you set this to a higher value, the throughput of storage operations gets better too.
The "storage-policy" config file setting tells MailerQ what type of messages should be stored in the message store. Valid values are "all", "out", "in" and "none". The "none" setting is meaningful if you only want MailerQ to retrieve mime data from external storage, without ever starting storage operations.
Before MailerQ publishes a message to RabbitMQ (for example, before it sends a received message to the inbox queue, or before it send a delayed message back to the outbox queue) it checks the storage policy to see whether the mime data should be sent to RabbitMQ too, or whether the mime data should be stored in a different storage system.
If you want all messages to be stored in the message store, use the "all" policy. If this policy is enabled, MailerQ checks each JSON object before it gets published to RabbitMQ. If the JSON still contains mime data, MailerQ removes this data from the JSON and stores it in the message store instead. The JSON data will be updated with a "key" property that refers to the data in the message store.
The "in" and "out" policies are more complex. The "out" policy instructs MailerQ to use the message store for outgoing messages only. If a message is greylisted or delayed and is published back to the outbox, MailerQ first strips the mime data from the JSON, and stores that in the message store. Incoming messages (like the ones that come in on the SMTP port, or the messages dropped in the spool directory) are not checked and the full mime data is published to RabbitMQ.
The "out" policy is often used, because most emails get delivered at the very first attempt, and it is therefore often a waste of resources to store incoming messages first in a NoSQL environment: the messages will probably be retransmitted a fraction of a second later. By using the "out" storage policy, initial injected emails are completely sent to RabbitMQ. Only if the initial delivery fails and the message is sent back to the outbox for later delivery, the full MIME data is stripped from the JSON and stored in the separate storage.
The "out" policy especially makes sense in setups where the majority of all deliveries succeed at the first attempt, and rescheduled attempts are likely to be pumped around between MailerQ and RabbitMQ for a number of times.
Only "all", "out" and "none" are meaningful policies. For completeness however, we also support the "in" property which does exactly the opposite as the "out" policy: all incoming messages are stored in the message store, and only metadata is published to RabbitMQ. However, when a mail is delayed and has to be published back to the outbox, the mime data is kept inside RabbitMQ.
When you store messages, you probably don't want to keep them forever in your message store. To overcome this, every message has a time-to-live value, and expired message are automatically removed from storage. The "storage-ttl" config option specifies the default time-to-live that is used for message that are stored in the document store.
Note that the time-to-live is added to the mail max delivery time. If you try to send out an email using MailerQ, and that email has to be delivered within 24 hours, and your "storage-ttl" is set to 3600 seconds (one hour), the mime data will be stored in NoSQL for at most 25 hours.
MailerQ uses a timeout for retrieval operations. If the storage server does not send back the message body within this time, the message is published back to RabbitMQ and will be retried a little later.
storage-timeout: 20
storage-reschedule: 120With the above settings you tell MailerQ to timeout fetch-operation after 20 seconds, and publish the mail back to RabbitMQ. After 120 seconds the message will be loaded again from RabbitMQ, and is retried.
You may choose to compress the messages you store to reduce the load on your storage server and limit the amount of data you need to send. This feature can be turned on by specifying "compression=gzip" in the storage URL, e.g.:
storage-address: mongodb://hostname/database/collection?compression=gzipMailerQ then takes care of compressing all data behind the scenes. Note that this does mean that the contents of your database are no longer human readable.
The compression feature is currently enabled only for
Support for the couchbase and SQLite backends is still in an experimental state.
MailerQ can open one or more SMTP ports on which it accepts incoming mail. All these incoming messages are published to RabbitMQ. The incoming messages are published to the inbox queue, as set with the "rabbitmq-inbox" setting in the config file.
Remember that MailerQ sends out message from the outbox queue - which is a different queue than the inbox queue to which incoming messages are published. By default, incoming message are therefore not immediately sent out again. To overcome this, most MailerQ users give the "rabbitmq-inbox" setting the same value as the "rabbitmq-outbox" setting. All incoming emails are then automatically published to the outbox queue, from which they are then immediately picked up again and scheduled for immediate forwarding.
However, you can also configure MailerQ to use different inbox and outbox queues. MailerQ then stores all incoming messages in the inbox queue first. You can add your own scripts that process these messages and forwards them to the outbox queue. Would you like to add a script that does additional processing or filtering before an incoming message is forwarded? Configure MailerQ to publish received messages in an inbox queue and let your scripts read these messages from this inbox queue. After processing, post the message to the outbox queue where MailerQ picks them up to deliver them.
If you use MailerQ to handle incoming email, you can also use scripts that pick up messages from the inbox queue, and store them in mailboxes or maildirs.
In the config file there are a number of variables that you can use to set the ports on which MailerQ should listen, and on which IPs MailerQ should accept incoming SMTP connections:
smtp-ip: 1.2.3.4
smtp-port: 25,2525,1024-1026
smtp-secure-port: 2526The "smtp-port" variable holds the port or ports that MailerQ opens and on which incoming connections are accepted. You can assign a single port, but comma-separated values and port-ranges are also accepted. The default SMTP port is 25, which is the one that you probably want to use.
Normally, MailerQ opens this port on all IP address that are available on the server. This means that if a server has multiple IP addresses, it does not matter to which of its IP addresses you connect: MailerQ listens on all of them. If you want to limit this, you can assign an explicit IP address using the "smtp-ip" variable. If you set this, MailerQ only accepts incoming connections to that specific IP.
The "smtp-port" setting contains the port number for the normal SMTP protocol. The SMTP protocol starts as a non-secure connection. The client and server can start a STARTTLS handshake to secure the connection. Besides normal SMTP connections, it is also possible to open an already secured SMTP socket. Use the "smtp-secure-port" setting for this. A secure port uses TLS right from the start, and no STARTTLS handshake is necessary. This is slightly faster (the initial handshake can be skipped) and is also more secure (the initial EHLO message can not be intercepted). However, such encrypted connections are not part of the SMTP standard, and regular SMTP clients do not expect this. However, if you write your own SMTP handshake code, it sometimes is simpler and faster to have access to a connection that is already encrypted without using "STARTTLS".
The IP address to which you send a message to MailerQ, is normally the same as the address from which the mail is going to be forwarded. Thus, if you send an email to MailerQ listening on IP address 5.6.6.5, the message will also be sent out from this IP. If you want to send out the message from a different IP instead, you should either add a special MIME header to your mail ("x-mq-ip"), or you can use the following config file variables:
smtp-default-ips: 1.2.3.4;4.5.6.7
smtp-unmappable-ips: 1.2.3.28/32;1.2.3.5/32
smtp-mappable-ips: 1.2.3.0/24;4.5.6.0/24The above config file settings tell MailerQ that the default outgoing IP addresses are 1.2.3.4 and 4.5.6.7. All messages that have not an explicit IP address from which they should be sent, will be sent from one of these two IP addresses.
Messages that are submitted over SMTP, are sent out from the same IP address as to which they were submitted if the IP address is mappable. Thus, if you send a message to 1.2.3.100, it will also be sent out from IP addresses 1.2.3.100. An IP address is considered to be mappable if it matches with the IP ranges inside the smtp-mappable-ips variable and does not match the ranges in the smtp-unmappable-ips variable. Thus, messages submitted to 1.2.3.100 are also sent out from 1.2.3.100 because that IP is mappable, while messages to 1.2.3.28 get sent out from the default IP's, because this IP address is not mappable.
To secure your SMTP traffic, you can assign a private key and certificate. You can use a self-signed certificate, but it is much better to use a private key and a certificate from a certificate authority. However, using a self-signed key is still better than using no key at all, as most SMTP clients will still accept the self-signed key.
smtp-certificate: /etc/mailerq/your.domain.com.crt
smtp-key: /etc/mailerq/your.domain.com.key
smtp-ciphers: !aNULL:!eNULL:!LOW:!SSLv2:!EXPORT:!EXPORT56:FIPS:MEDIUM:HIGH:@STRENGTHThe paths to the certificate and private keys for your domain should be added to the config file, with the list of ciphers that you'd like to support.
By default, no one can connect to the inbound SMTP server. With the "smtp-range" setting you can set the IP addresses from which you want to allow incoming email.
smtp-range: 192.168.0.0/16You can assign a semicolon separated list of IP ranges to the "smtp-range" variable.
In addition to specifying IPs you can also set an authentication string in the config file, all inbound connections must first authenticate before they can inject emails. For authentication to work, secure connections have to be configured first.
There are many possible ways to let users authenticate to MailerQ. MailerQ supports LDAP, RADIUS, linux users, HTTP endpoints, username/password files and lastly custom scripts. Keep in mind that if you set a username and password, the SMTP handshake becomes a little slower (incoming connections first have to authenticate before a message can be delivered). If you do not have to, it is therefore ofter faster to set up a firewall or an IP range to restrict access, than to require authentication.
smtp-auth: hardcoded://username:password,username2,password2
smtp-auth: ldap://example.com:389/base_dn
smtp-auth: radius://identifier:secret@example.com:1812
smtp-auth: users://user1,mailerq,user2,client*
smtp-auth: http://example.com:80/some_endpoint
smtp-auth: file:///users.txt
smtp-auth: exec:///path/to/fileThe first hardcoded is simply a comma separated list of username:password pairs, separated
by the ':' character. Note that this is the least safe, since the passwords will be plainly
readable in the config file, in unhashed form.
The ldap specifier will connect to the LDAP server and try to bind to DN=uid=username,base_dn.
If this succeeds, the user is authenticated, if this fails, the user is rejected.
The radius specified will simply request access to the radius server, with the shared secret secret and the identifier identifier. Make sure that the
client is set up correctly, or the server will reject the packets and MailerQ
logins will fail.
The http will perform HTTP BASIC authentication to the endpoint, and will therefore send along the
Authorization: Basic ... header. Any 2xx code is treated as a success, while any other code is
treated as a failure and a rejection.
The file specifier will load a simple text file that contains the usernames and passwords separated
by the : character, but with the passwords in hashed form (similar to /etc/passwd). These passwords
can be generated using mkpasswd -m sha-512 or openssl passwd -6, for example.
Lastly, the exec specifier will run the /path/to/file in a subprocess, and will check the return
code. This way, you can easily implement any authentication you require in a script! For example, you
can check a username and password combination in an SQL database, or you can ues the linux PAM modules,
or you can make very custom requests. This allows you to tie the SMTP authentication directly into
your existing user management.
If you have a username and password installed, messages from unauthenticated connections are normally rejected. However, if an incoming message is sent to an address that is marked as a local address, the message is accepted anyway. MailerQ checks each incoming message and compares the email address with the addresses listed on the "local address list". This list can be managed from the management console.
In the config file you can also assign a special message queue for such local email messages. Every time a message is accepted that is meant for a local address, it will be published to the queue that is assigned to the "rabbitmq-local" config file variable.
If you have also set a "rabbitmq-reports" variable in the config file, MailerQ parses all incoming local messages, and if it detects that it holds a report message (for example a Delivery Status Notification or a DMARC report), it will post the message to this reports queue.
MailerQ can run SPF, DKIM, DMARC and spam checks on incoming messages. The results of these checks are added to the JSON that is published to the inbox queue. You can also instruct MailerQ to modify the incoming messages and add an extra "authentication-results" header to the mail.
smtp-check: spf,dkim,dmarc,spam
smtp-auth-results: local,nonlocalThe "smtp-check" variable in the config file can be set to a comma separated list of checks that have to be performed on incoming messages. Supported values are "spf", "dkim", "dmarc" and "spam". The results of the checks are added to the JSON.
If you also want to add the check results to the "authentication-results" field in the MIME header, you can use the "smtp-auth-results" config file variable. This variable can be set to a comma separated list holding the possible values "local" and "nonlocal". If you set it to "local", the authentication-results header will only be added to incoming emails that were sent to a Local Email Address. The "nonlocal" value does exactly the opposite.
The DKIM, DMARC and SPF checks are all performed by MailerQ ifself. MailerQ has its own embedded algorithms to test whether incoming messages are valid. In practice, it comes down to extracting information from the e-mail, and doing a couple of DNS lookups to check whether the information in the mail matches the settings published in DNS.
Besides comparing mails with settings in DNS, MailerQ can also run a spam-check on incoming messages. For this, MailerQ uses the open source "SpamAssassin" application. Each incoming message is sent to SpamAssassin to test whether it is spam or ham. If you add the "spam" option to the "smtp-check" config file variable, you therefore also need a running SpamAssassin daemon (which is normally called "spamd"). The address where this daemon can be reached must be set in the config file.
spamassassin-host: localhost
spamassassin-port: 783The above config file settings tells MailerQ that the the SpamAssassin daemon is running on the same machine as MailerQ, and that it can be reached on port 783 (which is SpamAssassin's default port).
It is possible to run MailerQ behind a HAProxy server. If you do this, external clients do not directly connect to MailerQ, but to the proxy server instead. This proxy server then forwards the incoming traffic to one of the MailerQ backend servers.
All incoming traffic is thus first passed through the proxy server. From the perspective of MailerQ, the remote IP address of each incoming TCP connection is the IP address of the proxy, and not the (much more interesting) client address. To allow MailerQ to see the client IP address too, the PROXY protocol can be enabled on the HAProxy server. By enabling this protocol, the proxy server adds a small header in front of all forwarded TCP connections, with some meta information of the connection (most importantly, the client IP address).
With the "smtp-proxy" setting you tell MailerQ that when a connection comes in from a specific IP address, it should be treated as a connection from a proxy server, and the first bytes that are received should not be treated as the SMTP protocol, but as the initial PROXY header:
smtp-proxy: 192.168.0.0/16The "smtp-proxy" settings accepts a list of semicolon separated IP ranges. All incoming connections that match one of these ranges will be treated as proxied connections, and the first bytes will be treated as PROXY header, and not as SMTP traffic.
For the JSON, email address validation can be loosened,
so that attempts will be made despite an email address not being fully RFC compliant. To allow
injection of such messages, the smtp-validate-address option should be set. This option is for the
email supplied at the MAIL FROM and RCPT TO stage. Valid values are strict and loose.
By enabling injection for invalid emails, yet requiring RFC compliant emails during the sending phase, the mails can be initially accepted and processed by the rest of the infrastructure via RabbitMQ, instead of having to acount for the possibility that the injection goes wrong.
smtp-validate-address: strictTo prevent that MailerQ exhausts the max number of open TCP connections that is allowed by the OS, you can set a "smtp-connections" setting. There will never be more than this number of total incoming and outgoing connections.
smtp-connections: 100
smtp-connections-in: 50
smtp-connections-out: 75To prevent an excessive amount of either outgoing or incoming connections to block their counterparts from creating connections, the limits can also be adjusted separately. In the example above, the maximum amount of incoming connections are set to 50, but the maximum amount of outgoing connections are 75. When MailerQ is mostly sending, that means that MailerQ only allows 75 outgoing connections at the same time, which means that there are 25 connections reserved for the incoming connections. This guarantees that always at least 25 clients can be connected at the same time. If sending slows down, this number can grow up to 50, at which point it hits the incoming connection limit, preventing it from taking up the connection space for outgoing connections.
If only the global limit would be set in the example above, if MailerQ is connected to 100 endpoints, incoming connections would be queued until the connection limit dropped again. If there are already 100 incoming connections, MailerQ would be unable to send any email, because the global connection limit is already exhausted.
Usually, you want all three values to be set, to guarantee that ever side has a reserved space for the amount of connections that they can make.
The following variables may also be useful when you set up an SMTP server:
smtp-maxsize: 100MB
smtp-threads: 1
smtp-timeout: 300
smtp-protection: true
smtp-hostname: smtp.example.comIn the initial SMTP handshake the client advertises its capabilities (like the max message size to accept). The config file setting "smtp-maxsize" contains the maximum allowed input size for message.
All SMTP traffic is handled by separate threads. The number of threads that are started for SMTP traffic can be set with the "smtp-threads" setting. Increasing this value can give a real boost to MailerQ's performance and we recommend to set it close to the number of CPU's that you have in your machine.
Important! The "smtp-threads" variable is meaningful for outgoing connections as well!
When a client connects to the SMTP server but stops responding before the transaction is complete, MailerQ will wait for a response for the amount of seconds set in "smtp-timeout"
To protect your incoming SMTP server, you can set a list of IP addresses that you suspect of abuse. "smtp-protection" enables blocking of these addresses when they connect.
A client that connects to the SMTP server, will be greeted with a message containing a hostname. This normally is the hostname that is associated with the MTA and is, per IP, configurable. The hostnames are normally auto-detected via reverse DNS or can be overridden via the management console. If the inbound MTA is not yet associated with an MTA, the setting from the config file is used. If the config file also holds no setting, a reverse DNS lookup is started. So the algorithm for finding the hostname used in the SMTP welcome message is:
If you have not used the "smtp-ip" setting and you run MailerQ on a server with multiple IP addresses, the SMTP server is available on all the server's IP addresses. A client can therefore choose to which IP address to send your mail. MailerQ recognizes the IP address to which the mail was originally submitted, stores that information in the JSON message, and when the mail is finally forwarded to the internet, it will be sent out from exactly the same IP address. You should be aware that this can cause problems if you deliver your email to an IP address from which no external connections can be made (like 127.0.0.1).
If you want MailerQ to send out the mail from a different IP address than that you originally sent it to, you can include an extra MIME header field that instructs MailerQ to use a different IP instead.
All incoming messages are converted into JSON messages and published to RabbitMQ. The minimal JSON message contains the envelope address, the recipient and the mime data, which is all extracted from the SMTP protocol.
If you want to add more settings to the JSON, you can do so with special headers in the MIME message. Every header that starts with "x-mq-" is treated as a special MailerQ header, and will be turned into a JSON property that controls how the mail is going to be handled.
This feature to automatically extract meta data from the MIME header can be disabled in the config file. If you use MailerQ as an internet facing receiver, you probably do not want to allow users to control the deliverability by adding "x-mq-*" these headers, so in that case you can better disable this feature in the config file:
smtp-extract: true (default: true)The mq_cleanup() function is an optional function that you could implement in a plugin. When MailerQ is about to shut down , it will check every plugin to find out if this mq_cleanup() function is available, and if it is, it will call it.
You can use this function to run your own cleanup code.
#include <mailerq.h>
/**
* Clean up stuff
*/
MQ_EXPORT void mq_cleanup() {
// @todo add your cleanup code
}Set whether this connection should be considered authenticated. Note that if a previous plugin had already set this connection to be authenticated and you set it to be unauthenticated, the login will be rejected.
/**
* Set whether the connection should be considered authenticated
*
* @param connection the connection that authentication should be set for
* @param bool should the credentials be considered valid
*/
void MQ_setAuthenticated(MQ_Connection *connection, bool authenticated);If you want to change the MIME data that is associated with a MQ_Message, you can use this function. This function sets the MIME data.
/**
* Update the MIME data
*
* @param message the message to set the envelope address in
* @param mime actual mime string
* @param size length of the mime string
*/
void MQ_setMime(MQ_Message *message, const char *mime, size_t size);This function has a different behavior if you call it on a message that is being received and on a message that is being sent. If you call it on a message that is being received, for example inside your mq_smtp_in_message() function, it is simply an alias for setting the "mime" property in the JSON. Changing the MIME by calling this function, will also change the JSON that is going to be published to RabbitMQ.
For outgoing messages however, changing the MIME data with this function will not modify the JSON data. If your plugin calls this function before the actual data is sent over the SMTP connection, the newly set data is going to be used. However, when the message is published back to RabbitMQ (for example to the result queue, or back to the outbox queue for a retry) it will still use the original JSON object. If you also want to set "mime" property in the JSON, you should use the MQ_json() function as well.
This function is often used in plugins that implement the mq_smtp_out_data() function.
To integrate MailerQ into your PHP environment, you can either write PHP scripts that inject emails directly into RabbitMQ (using the AMQP protocol), send mails to MailerQ over SMTP, or change the php.ini config file so that the builtin PHP "mail()" function automatically injects mail into MailerQ.
The PHP mail() function internally uses a command line utility (by default "sendmail") to send out the mail. Every time that you make a call to this mail() function, the PHP engine starts up this command line utility and sends out the message. By modifying the "php.ini" file you can change the command line utility into "mailerq" so that all mails sent from PHP are automatically delivered to MailerQ.
To use MailerQ for sending emails in PHP using mail() function, set the "sendmail_path" property in the php.ini config file to:
sendmail_path = mailerq --envelope my-sender-address@my-domain.com --extract-recipients --ignore-dot
Despite the simplicity of the above solution, it is more efficient to inject mails directly into RabbitMQ. In the following example we will demonstrate you how to access RabbitMQ from PHP using the AMQP protocol. This example assumes that MailerQ is running and configured and that the PECL AMQP package extension is installed.
Read more about the AMQP protocol
The example consists of three files; settings.php, send.php and result.php.
If you want to use these scripts, save the three files inside the same folder.
The settings.php file defines the settings that are needed to connect to
RabbitMQ and it also contains the settings that are being used for sending
an email. If you want to use the PHP example to test your MailerQ configuration,
you will have to adjust the values inside the settings.php file to match your
configuration of MailerQ.
<?php
/**
* This file holds the settings that the PHP test script uses to connect
* to RabbitMQ and to construct the message that will be placed on the
* outbox message queue.
*/
$address = 'amqp://guest:guest@localhost'; // same as the rabbitmq-address option in the config file
$outbox = 'outbox_test'; // same as the rabbitmq-outbox option in the config file
$resultbox = 'results_test'; // same as the rabbitmq-results option in the config file
// email message settings
$recipientDomain = 'example.org'; // domain where the test message should be delivered
$recipientEmail = 'info@example.org'; // email where the test message should be delivered
$fromAddress = 'me@my-domain.com'; // address where the email was sent from
?>
This PHP script connects to RabbitMQ and places a JSON encoded message on
the outbox message queue. To connect to RabbitMQ and to create the JSON encoded
message, the script uses the values from the settings.php file. If you want
to use this script to test your MailerQ configuration, you will have to run
the send.php script before running the result.php script.
<?php
/**
* send.php
* Script that connects to RabbitMQ, constructs a json encoded message
* and puts that message on the outbox message queue.
*/
// include the settings
require_once('settings.php');
// try to set up a connection to the RabbitMQ server / try to log in
try
{
// construct the connection to the RabbitMQ server
// could add vhost, port as well
$connection = new AMQPConnection(array(
'host' => $hostname,
'login' => $username,
'password' => $password
));
// connect to the RabbitMQ server
$connection->connect();
}
catch (AMQPException $exception)
{
echo "Could not establish a connection to the RabbitMQ server.\n";
}
// try to create a channel for sending instructions
try
{
// create the channel
$channel = new AMQPChannel($connection);
}
catch (AMQPConnectionException $exception)
{
echo "Connection to the broker was lost (creating channel).\n";
}
// try to declare the exchange, which will function as an outbox were
// rabbitmq will find the messages that need to be sent
try
{
// declare the exchange
$exchange = new AMQPExchange($channel);
$exchange->setName('exchange1');
// this type sends to all queues connected to this exchange
$exchange->setType('fanout');
$exchange->declareExchange();
}
catch (AMQPExchangeException $exception)
{
echo "Channel is not connected to a broker (declaring exchange).\n";
}
catch (AMQPConnectionException $exception)
{
echo "Connection to the broker was lost (declaring exchange).\n";
}
// create the queue mailerq will get its messages from,
// which will get those from the exchange
try
{
// create the queue
$queue = new AMQPQueue($channel);
$queue->setName($outbox);
// queue survives rabbitmq restart
$queue->setFlags(AMQP_DURABLE);
$queue->declareQueue();
// route from exchange1 to this queue;
// the routing key is optional and what it does depends on the
// exchange type. fanout ignores it
$queue->bind('exchange1','key2');
}
catch (AMQPQueueException $exception)
{
echo "Channel is not connected to a broker (creating queue).\n";
}
catch (AMQPConnectionException $exception)
{
echo "Connection to the broker was lost. (creating queue)/\n";
}
// JSON encoded message to put on the queue
$jsonMessage = json_encode(array(
'envelope' => $fromAddress,
'recipient' => $recipientEmail,
'mime' => "From: " . $fromAddress . "\r\n"
. "To: " . $recipientEmail . "\r\n"
. "Subject: Example subject\r\n\r\n"
. "This is the example message text"
));
// try to publish the message on the queue. fanout sends to all, key1 is used as filter
try
{
$message = $exchange->publish($jsonMessage, 'key2');
}
catch (AMQPExchangeException $exception)
{
echo "Channel is not connected to a broker (publishing message on queue).\n";
}
catch (AMQPConnectionException $exception)
{
echo "Connection to the broker was lost (publishing message on queue).\n";
}
catch (AMQPChannelException $exception)
{
echo "The channel is not open (publishing message on queue).\n";
}
// done, close the connection to RabbitMQ
$connection->disconnect();
?>
This PHP script connects to the RabbitMQ server, and gets the message that was
placed on the outbox message queue by the send.php script back from the result
message queue. The result message from the result queue is shown to the user.
The result.php can only output any relevant information when it is executed after
there is something to show in the results queue, for example by executing the send.php script.
<?php
/**
* result.php
* Script that connects to RabbitMQ, and takes all messages from
* the result message queue. This could be useful when you want to
* parse messages from a custom queue and later re-post them, for example
* to the outbox queue.
*/
// include the settings
require_once('settings.php');
// try to set up a connection to the RabbitMQ server
try
{
// construct the connection to the RabbitMQ server
$connection = new AMQPConnection(array(
'host' => $hostname,
'login' => $username,
'password' => $password,
'vhost' => $vhost
));
// connect to the RabbitMQ server
$connection->connect();
}
catch (AMQPException $exception)
{
echo "Could not establish a connection to the RabbitMQ server.\n";
}
// try to create the channel
try
{
// open the channel
$channel = new AMQPChannel($connection);
}
catch (AMQPConnectionException $exception)
{
echo "Connection to the broker was lost (creating channel).\n";
}
// try to create the queue
try
{
// create the queue and bind the exchange
$queue = new AMQPQueue($channel);
$queue->setName($resultbox);
$queue->setFlags(AMQP_DURABLE);
// again with the optional routing key
$queue->bind('exchange1', 'key2');
$queue->declareQueue();
}
catch (AMQPQueueException $exception)
{
echo "Channel is not connected to a broker (creating queue).\n";
}
catch (AMQPConnectionException $exception)
{
echo "Connection to the broker was lost. (creating queue)/\n";
}
// Get all messages from the queue.
while ($envelope = $queue->get()) {
echo "Received:\n";
echo $envelope->getBody() . "\n";
echo "----------------------------------------------\n\n";
}
// done, close the connection to RabbitMQ
$connection->disconnect();
?>
MailerQ’s messaging system includes support for delayed message handling — a feature critical for scenarios such as managing graylisting requirements, enforcing send limits, or staging messages for later delivery. RabbitMQ, however, is fundamentally a FIFO (First-In, First-Out) system, not designed with inherent delay capabilities. Messages sent to a RabbitMQ queue are processed as soon as they are available to consumers, making it challenging to hold messages back for a specific period.
To work around these limitations, MailerQ employs a solution by leveraging RabbitMQ’s dead-letter exchanges and message timeouts to create the effect of delayed message processing. This article covers the configuration, functionality, and limitations of this delay mechanism, along with an alternative solution using the amqpdelayer application for more efficient delay management.
MailerQ implements delays by routing messages to a series of “delay queues,†each configured with a specific timeout. Here's how this process works in steps:
Parking messages in dead-end queues: When MailerQ determines that a message requires a delay, it places the message in a “dead-end†queue — a queue that is not consumed by any application and thus theoretically holds the message indefinitely. However, this queue has a timeout setting, so once the specified time elapses, RabbitMQ automatically expires the message.
Dead-letter routing: Each delay queue is also set up with a dead-letter exchange and routing key that directs expired messages back to MailerQ. When the message timeout is reached, RabbitMQ automatically forwards the expired message to MailerQ via the dead-letter exchange. MailerQ then decides if the message delay period has elapsed. If further delay is required, MailerQ moves the message to a new delay queue with a shorter timeout. This process continues across several delay queues until the delay period has ended. Finally, when no more delay is needed, the message is routed back to the outbox for processing.
Powers-of-two queue system: MailerQ sets up delay queues based on powers of two (1 second, 2 seconds, 4 seconds, etc.) to enable flexible delay durations. For example, if a message needs a 10-minute (600-second) delay, MailerQ places it first in a queue with a 512-second timeout (the largest power of two under 600), followed by a 64-second queue, and so on until it reaches the outbox with no delay.
The delay queues and bindings are automatically created when MailerQ starts. If required, settings in the MailerQ configuration can be modified to control the behavior and structure of these queues:
These configurations allow MailerQ to manage and customize delay queues, supporting flexible delay times while maintaining control over queue durability and memory usage.
Using delay queues within RabbitMQ has several benefits and drawbacks:
Integrated and simple setup: The delayed queue system is part of RabbitMQ and MailerQ’s configuration. When MailerQ starts, it automatically creates the necessary delay queues, and messages are managed without additional infrastructure.
Reliable delays: By setting exact timeouts in seconds, MailerQ can ensure that delayed messages appear in the outbox queue at the desired time, meeting application-specific scheduling needs.
Resource-intensive: Every delay cycle involves messages expiring, being forwarded to the next queue, and eventually reappearing in the outbox. This increases the workload on RabbitMQ, MailerQ, and the network, particularly with frequent or high-volume delays.
Not designed for long-term storage: RabbitMQ is optimized for real-time message flows, not for long-term message storage. Using RabbitMQ to hold messages for extended periods can affect performance.
Queue complexity: The additional delay queues make the queue topology more complex, especially with multiple queues for various timeouts. This may complicate queue management and monitoring.
For high-load or longer delays, an alternative approach is using amqpdelayer, a custom application designed to handle delayed messages outside of RabbitMQ. It reads delayed messages from RabbitMQ and holds them temporarily within its own storage until the delay period is complete. After the specified delay, it sends the message back to RabbitMQ for final processing, significantly reducing the workload on RabbitMQ by minimizing repeated message forwarding and queue cycling.
Since amqpdelayer stores messages independently, RabbitMQ’s delay queues don’t experience the constant expiry and movement of messages, resulting in a lighter load on RabbitMQ and MailerQ and improved overall efficiency.
Integrating amqpdelayer into your mail stack requires adjustments to the RabbitMQ topology. The goal is to route all retries and reschedules to amqpdelayer, and after their delay back to the outbox. This can be achieved in many ways, but our recommendation is to setup an extra exchange and queue, while keeping the same routing keys. This gives the best monitoring and debugging options.
Separate exchanges for applications: Give each application its designated exchange. MailerQ writes to the "mailerq" exchange, while amqpdelayer writes to a separate "amqpdelayer" exchange.
Separate queues for applications: Each application gets its own input queue. MailerQ reads its input from the "outbox" queue, and amqpdelayer from the "delays" queue.
Routing delayed messages to amqpdelayer: The normal bindings between the "mailerq" exchange and the "outbox" queue should be removed, and replaced with two similar bindings between the "mailerq" exchange and the "delayed" queue. Add a binding for the routing key "outbox" and one for routing key "retries".
Routing processed messages to mailerq: Add two new bindings from the "amqpdelayer" exchange to the "outbox" queue. Use the routing key "outbox" and "retries" here too.
The above changes ensure that messages are still being moved around with the same routing keys, but are first processed by amqpdelayer before they are returned to MailerQ.
Disable queue declaration: Because of the custom topology, you can no longer rely on MailerQ setting up the topology. The variables rabbitmq-declare and delayed-declare should therefore both be set to false.
Reduce number of delayed queues: MailerQ no longer has to pump messages around to the delayed queues, so the setting delayed-queues can be set to 0.
The config file of amqpdelayer should be modified to make it read from the "delayed" queue, and publish to the "amqpdelayer" exchange, while preserving routing keys. You especially need to pay attention to these settings:
The variable rabbitmq-exchange in the amqpdelayer config file should be set to amqpdelayer.
The variable rabbitmq-routingkey should be empty. This will make amqpdelayer publish the output messages with their original routing key. This can be helpful for debugging.
If the default behavior of MailerQ is not up to your standards, you can extend MailerQ with your own plugins. MailerQ allows you to develop plugins to add features or functionality to MailerQ. A plugin is a shared object file (like "example.so"), and normally developed in C or C++. The plugins are loaded by MailerQ when the application starts. Every plugin can implement a number of functions that are called by MailerQ to hand control over to the plugin.
In the global configuration file, you can set a "plugin-directory" variable. This variable should hold the name of the directory where MailerQ can find your plugins. The plugins are loaded in alphabetical order. If multiple plugins are loaded, MailerQ will make the calls to the plugins in this same alphabetical order. If the order in which your plugins are called matters, you can prefix your plugin name with a number, like "10-example.so" and "20-anotherexample.so". By doing this, you ensure that the "example" plugin will always be called before the "anotherexample" plugin.
MailerQ makes the calls to your plugin using the C calling convention. As a
result, it is best and easiest to use C or C++ for developing your plugin. When
developing in C++, be sure to extern "C" {} the function which will be called
by MailerQ, to prohibit the compiler from function name mangling.
Plugins are called often -- possibly multiple times for every message. You should be aware of this and write your plugins with care. Blocking IO calls are for example absolutely a no-go as they will massively bring down MailerQ's performance.
To find out how MailerQ hands over control to your plugin, how your plugin can then interact with MailerQ's event loop, and finally give back control to MailerQ, please read our event loop article.
The "mq_version()" function is currently the only function that your plugin is required to implement. All other callbacks are optional. A very basic plugin (one that does nothing) would look like this:
#include <mailerq.h>
/**
* Function called by MailerQ to find out if the plugin is compatible
* with the MailerQ executable
*
* @return int API version number for which the plugin is compiled
*/
MQ_EXPORT int mq_version() {
return MQ_API_VERSION;
}
/**
* Function that is called by MailerQ to initialize the plugin. This
* function is called right after the plugin is loaded.
*
* This is an optional function, you do not have to implement it. MailerQ
* only calls it if it exists.
*/
MQ_EXPORT void mq_initialize() {
// @todo add your own initialization code
}
/**
* Function that is called right before MailerQ stops running.
*
* This is an optional function, you do not have to implement it. MailerQ
* only calls it if it exists.
*/
MQ_EXPORT void mq_cleanup() {
// @todo add your own cleanup code
}The "MQ_EXPORT" directive is a C #define that is turned by the preprocessor in
a statement to ensure that the symbol is correctly exported. If you compile your
extension with gcc, it is for example turned into
__attribute__ ((visibility("default"))).
When compiling your extension, we recommend using hidden visibility for all functions not to be made available externally (all functions that MailerQ won't call). Doing this gives the compiler much more freedom to inline functions (which it cannot do if a function is to be made available externally) and helps keep the DSO small.
For our SMTPeter cloud service we use MailerQ for handling incoming messages via SMTP. For every incoming connection we check the SMTP login credentials, and compare them with the login data that we have on file. Only recognized users are given access.
We use a MailerQ login to handle these incoming login requests. A simplified version of this plugin is available as example to see how such a plugin can be built.
If you have the idea that vital functionality is missing in the plugin API, or if you need other callbacks you can always get in contact with us. We can accomodate your needs, extend the API and implement your request!
This function stops a timer that was previously created. Note that this function should only be called to stop a timer that has not yet fired yet, because a timer will automatically be cleaned up and calling MQ_stopTimer after this is not legal.
/**
* Stop a timer before it expires
*
* @param timer the timer to stop
*/
void MQ_stopTimer(MQ_Timer *timer);
For a detailed example, see MQ_startTimer
MailerQ is configured via one central configuration file: "/etc/mailerq/config.txt". It holds many options. The most important ones are the address and login credentials of your RabbitMQ message broker and the address of your database. You probably need to change these settings. All other config file settings have decent defaults that work directly (although you might want to take a look at them later).
The "rabbitmq-address" setting in the config file holds the address of the RabbitMQ server that you want to use for message queueing. It has the format "amqp://user:password@hostname/vhost". The default value ("amqp://guest:guest@localhost/") works if your RabbitMQ server is running on the same machine as MailerQ, and when you've not altered the default guest/guest credentials.
rabbitmq-address: amqp://user:password@hostname/vhostIf you have a cluster of RabbitMQ nodes, the hostname can be separated by semicolons (e.g. amqp://user:password@host1;host2;host3/vhost). Setting up a RabbitMQ cluster means you're less likely to lose messages when one RabbitMQ server fails.
MailerQ stores all runtime settings in a relational database. This can be a MySQL, MariaDB, PostgreSQL or SQLite3 database. The SQLite3 database is by far the easiest one to set up, because it does not require a database server and all you have to do is just enter the path to a file on your server where MailerQ can store its runtime settings. This file does not even have to exist as it will be automatically created by MailerQ when it is missing.
database-address: sqlite:///path/to/database/file.sqlWhen MailerQ first connects to a database (or first opens the database file), it automatically creates all database tables and initializes everything.
To be able to connect, you do have to ensure that the appropriate database client libraries are installed on your system. If you use the sqlite database engine, make sure you have libsqlite3 on your system.
When an incoming SMTP connection is turned into a secure SMTP connection, MailerQ will call the mq_smtp_in_secure() function in the plugins. This function is called after the connection is secured, so you can be sure that the passed in connection object is guaranteed to be secure.
Your function should return true if you want your plugin to take over control from MailerQ, or false if control should stay with MailerQ.
bool mq_smtp_in_secure(MQ_Context *context, MQ_Connection *connection);
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
The mq_context_initialize() is an optional function that you can add to your plugin. It is called by MailerQ during startup for each worker thread that is created. This is useful if, for example, you want to set up a database connection per thread, or want to do other initialization.
This function can be implemented by the plugin and is called by MailerQ when a worker thread starts. This function cannot take control by returning boolean true, but since this function is executed only during startup, it is acceptable to block here. MailerQ will wait until all contexts are initialized before continuing.
#include <mailerq.h>
/**
* This function is called by MailerQ for every worker thread
* that is created
*/
MQ_EXPORT void mq_context_initialize([MQ_Context](mq_context) *context) {
// set up a connection to MySQL
MYSQL *connection = mysql_init(NULL);
// TODO: connect here and do other useful stuff
// store the connection so it can be used in later callbacks
MQ_setContextData(context, connection);
}If you allocate memory in this function, you probably also have to add a mq_context_cleanup to your plugin.
Get access to the socket filedescriptor that represents the underlying TCP connection of the SMTP connection.
Watch out! The socket is completely managed by MailerQ, and it is in general not a good idea to use the filedescriptor for sending or receiving data, or to close it. Your plugin should only use the filedescriptor to retrieve information about the connection, for example to retrieve the local or remote IP addresses of the connection using regular C functions getsockname() or getpeername().
/**
* Retrieve file descriptor that is used for the SMTP connection
* @param connection the connection in which the data was stored
* @return int
*/
int MQ_fileDescriptor(MQ_Connection *connection);
This function is used to reset the timeout for a given timer without having to destroy and recreate the timer. This function may not be called on a timer that has already expired.
/**
* Reset a timer
*
* @param timer the timer to reset
* @param timeout the new timeout for the timer
*/
void MQ_resetTimer(MQ_Timer *timer, float timeout);
For a detailed example, see MQ_startTimer
If you inject mails in SMTP format into MailerQ, you can add special headers to your message that are read and recognized by MailerQ, and that control the delivery of your message. Headers that start with "x-mq-*" are automatically recognized by MailerQ, stripped from the email message and converted into JSON properties before the message is sent to one of the message queues.
Almost all "x-mq-*" properties are converted into JSON properties, even custom properties that do not have a special meaning (like "x-mq-meaningless"). However, the following properties have a special meaning and have influence on the handling of your message.
| x-mq-envelope | explicit envelope address |
| x-mq-ip | IP address to send mail from |
| x-mq-pool | pool identifier to send the message from |
| x-mq-tag | adds a tag to the message |
| x-mq-delayed | time when message should be sent |
| x-mq-maxdelivertime | max time for delivery |
| x-mq-maxattempts | max number of delivery attempts |
| x-mq-retries | interval in seconds between the attempts |
| x-mq-inlinecss | instruct MailerQ to inlinize the CSS code |
| x-mq-results-queue | alternative queue for the results |
| x-mq-failure-queue | alternative queue for the failures |
| x-mq-success-queue | alternative queue for the successes |
| x-mq-retry-queue | alternative queue for the retries |
| x-mq-smarthost-hostname | hostname of smarthost to send email to |
| x-mq-smarthost-port | portnumber of the smarthost |
| x-mq-smarthost-username | login for the smarthost |
| x-mq-smarthost-password | password to access the smarthost |
| x-mq-dsn-notify | the events for which to receive a DSN |
| x-mq-dsn-ret | whether to send the full body or just the headers in DSN |
| x-mq-dsn-orcpt | the original recipient for the DSN |
| x-mq-dsn-envid | unique identifier for the DSN message |
| x-mq-keepmime | do not remove the full mime object after delivery |
The following message would be an example MIME message that holds some extra meta properties:
from: info@example.com
to: peter@smtpeter.com
subject: example message
x-mq-ip: 10.11.12.13
x-mq-maxdelivertime: 2016-10-23 23:00:00
x-mq-inlinecss: 1
x-mq-custom-identifier: my-id
this is the body of the mailWhen the message above is injected into MailerQ, it will be transformed into a JSON object that is sent to the inbox queue. The JSON object will have the following properties:
{
"envelope": "info@example.com",
"recipient": "peter@smtpeter.com",
"mime": "....",
"ips": [ "10.11.12.13" ],
"maxdelivertime": "2016-10-23 23:00:00",
"inlinecss": true,
"custom-identifier": "my-id"
}To ease readability, we have left out the mime data. The mime data would be identical to the injected mime, without all the x-mq-* headers.
The envelope address from an injected mail is extracted from the MIME message. MailerQ checks (in this order) the following properties to extract the envelope address: "x-mq-envelope", "return-path", "from" and "sender". The first valid email address to be found in one of these headers will be used as envelope address.
The envelope address is the address that is used in the "MAIL FROM" communication.
If your server has multiple IPs to send out mail from, you can use all of them to send out mail from. To control the IP address that is going to be used, you can use the "x-mq-ip" property.
To allow multiple IP address for sending out the mail, you can add a list of available IP addresses:
x-mq-ip: 231.34.13.156
x-mq-ip: 231.34.13.158
MailerQ will pick one of the IPs to send out the mail. Be aware that you can of course only use addresses that are actually bound to the host that MailerQ runs on. Other IP addresses will result in failed deliveries.
You may be injecting different types of messages into MailerQ, for example messages belonging to your various customers or campaigns. To help you get a better overview, MailerQ allows you to tag your messages with one or more labels of your choosing. These tags will show up in the management console, allowing you to monitor and control all deliveries belonging to a tag. To add a tag in a MIME email you can add one or more "x-mq-tag" properties to the header:
x-mq-tag: Customer Name
x-mq-tag: Example CampaignMailerQ will track messages belonging to a tag and allow you to monitor them in the management console. See the tags page for more information.
When a message cannot be delivered immediately because of unresponsive receivers, greylisting or throttling, MailerQ publishes back the email to the outbox queue for later delivery. This can result in emails that are sent much later than the time that you first added them to the message queue.
If you do not want an email to be delivered after a certain time, all you need to do is add the "x-mq-maxdelivertime" header to the message. The "x-mq-delayed" property does exactly the opposite: it delays the email so that it is not delivered before a specific time.
x-mq-delayed: 2016-02-10 12:00:00
x-mq-maxdelivertime: 2016-03-10 00:00:00Timestamps in MailerQ are always in UTC. MailerQ is not very tolerant in parsing timestamps, so better make sure that you use the right formatting (YYYY-MM-DD HH:MM:SS).
If you do not specify an explicit max delivery time, MailerQ will attempt to deliver the mail within 24 hours (default) after the mail was first picked up from the outbox. You can change the default in the configuration file.
Just like a maximum delivery time, you can control the maximum number of attempts MailerQ should make sending an email. If a first attempt fails because a remote server is unreachable or does not immediately accept the message, MailerQ will make a new attempt a little later.
x-mq-maxattempts: 3
If a mail is retried, MailerQ waits a while before the next attempt is sent. You can set this dely in the headers of the message.
x-mq-retries: 300,600,3600The above setting means that the second attempt is ent 300 seconds (5 minutes) after the first attempt failed. If the second attempt failed too, MailerQ waits another 600 seconds (10 minutes) for the next attempt. After that attempt, all further attempts will be sent with a one hour (3600 seconds) delay.
When you send out HTML emails, you face the problem that not all email clients
support stylesheets that are set in the header. Some email clients (especially
web based clients) strip out the CSS code from the HTML header. This often messes
up the layout and look of your messages. To overcome this, it is better not to set
CSS settings in the header in the first place, but use style="..." attributes
in the HTML code instead.
MailerQ can do this automatically. If you set the x-mq-inlinecss header to true,
MailerQ parses the HTML email, and converts the CSS code from the HTML header
into inline style="..." attributes in the HTML body.
x-mq-inlinecss: 1
After MailerQ has processed an email, it publishes the result of the delivery to one or more result queues. The default queues to use for this are configured in the global configuration file, but you can set other queue names in properties to tell MailerQ to use other result queues.
x-mq-results-queue: name-of-results-queue
x-mq-failure-queue: name-of-failure-queue
x-mq-success-queue: name-of-success-queue
x-mq-retry-queue: name-of-retry-queue
Note that when only one if these "x-mq-*-queue" properties is used, all the queues mentioned in the global configuration file are ignored. This is even so if you have not even specified all possible queues in the MIME header.
Instead of having MailerQ sending messages directly to the final recipient, it can be sent to an alternative mailserver instead: the smarthost. The default credentials can be set in the config file, or they can be provided the following properties:
x-mq-smarthost-hostname
x-mq-smarthost-port
x-mq-smarthost-username
x-mq-smarthost-passwordRead more about using a smarthost
Some mail servers initially seem to accept a message, but send back a bounce email later. Delivery Status Notification (DSN) messages help you keep track of what happened to your mails. The following options specify which Delivery Status Notifications you want to receive:
x-mq-dsn-notify
x-mq-dsn-orcpt
x-mq-dsn-ret
x-mq-dsn-envidRead more about Delivery Status Notifications
When a message is completely processed - either because it was successfully delivered, or the delivery failed - MailerQ publishes it to the results queue, where you can pick it up for further processing.
By default, MailerQ throws away the mime data to make room in the JSON object and
in the message store. If you do not want the message data to be removed, you can
tell so by adding the keepmime option:
x-mq-keepmime: 1
Once you have a running MailerQ instance, you want to use it to send out mailings. But how do you get your mails into MailerQ? There are many ways to do this:
After sending the email, MailerQ publishes a JSON-encoded copy to the result queue, and, depending on whether the delivery was successful, into the success or failure queues. Your own application or script can read these results up for further processing.
If you want to check whether a received incoming message was marked as a local message for local delivery, you can call this function. It returns boolean.
/**
* Retrieve the local setting
*
* @param message the message to retrieve the setting from
* @return the local setting
*/
bool MQ_local(MQ_Message *message);For more info, see the documentation about MQ_setLocal.
Available from MailerQ 5.6.5.
/**
* Log priorities
*/
#define MQ_CRIT 1
#define MQ_WARN 2
#define MQ_INFO 3
/**
* Log something in the MailerQ application-log. This will be prepended
* by the priority and the time in the final log entry.
* @param priority
* @param message
*/
void MQ_log(int priority, const char *message);This function is safe to be called from any thread simultaneously, and will log to the open MailerQ application-log output streams as set in the settings.
MQ_log(MQ_CRIT, "log-1");
MQ_log(MQ_WARN, "log-2");
MQ_log(MQ_INFO, "log-3");
MQ_log(-1, "log-invalid")If the plugin contains the snippet above, upon execution the log outputs as specified to the application log outputs. If an incorrect priority is passed, as above, (invalid priority) will be prepended to the log line and it will be logged under info priority.
[Critical] [2019-09-23 12:46:56] [plugin] log-1
[Warning] [2019-09-23 12:46:56] [plugin] log-2
[Info] [2019-09-23 12:46:56] [plugin] log-3
[Info] [2019-09-23 12:46:56] [plugin] (invalid priority) log-invalidEverything in MailerQ has a non-blocking nature. Things like network communications, database queries and timeouts have been implemented in a non-blocking fashion. This means that MailerQ does not block while it is waiting for an answer from a remote server or from a database, but that it continues with other tasks for as long as no answer is available.
In order to get most out of MailerQ and your plugin, all your plugin functions should be implemented in the same way, and should also respect this non-blocking nature. For example, if your plugin needs to download something, you should not make a blocking network call, because this will block the entire MailerQ process (or to be more precise: an entire MailerQ worker thread). You should be using the supplied MailerQ event loop instead.
Another thing to carefully consider is how many times your implemented function will run during the application lifetime, especially with regard to speed of your plugin. When you create the plugin, keep in mind that your implemented function will be called for every message and MailerQ may bottleneck on a slow plugin.
Every time MailerQ is processing or delivering messages, one or more of your plugin callback functions can be called. When a certain event occurs, MailerQ checks the loaded plugins in alphabetical order to see if a certain callback function is implemented in the plugin shared object file. If the function is indeed implemented, it will be called by MailerQ to hand over control to the plugin. For example, if three plugins "A", "B", and "C" are loaded and MailerQ is at the "DATA" state during a SMTP handshake, MailerQ will check all these three plugins to see if they have implemented the mq_smtp_out_data() function. For every plugin that has implemented the function, it will call that function. Plugins that have not implemented the function are skipped.
Plugin functions should return a boolean value. When a plugin returns "false", it means that the plugin does not take over control from MailerQ, and MailerQ can proceed with calling the next plugin, or if there are no other plugins, proceed with delivering or receiving the message. However, if your plugin returns "true", MailerQ will stop processing the message, and will continue doing other tasks. Your plugin now is in control, and will stay so until you hand the control back to MailerQ.
MailerQ passes a number of parameters to a plugin function. One of these parameters is a pointer to a MQ_Context. This MQ_Context pointer gives you access to the MailerQ event loop, and allows you register file descriptors and set timers. If your plugin wants to do a blocking network call, it can do this in a non-blocking fashion instead, and register the socket and a callback function. Your callback will be called as soon as the filedescriptor becomes active.
When your plugin is ready with its task - it has for example downloaded the appropriate resources, or has modified or filtered the message that is going to be delivered - it must give control back to MailerQ. It can do this in three possible ways, with one of the following methods:
| Function name | Description |
|---|---|
| MQ_continue() | Tell MailerQ to call the next plugin |
| MQ_complete() | Tell MailerQ to skip all further plugins |
| MQ_retry() | Tell MailerQ to restart calling all plugins |
After your plugin has done its work, it can tell MailerQ that it is ready and that the next plugin can be called (this is what normal plugins do), or it can instruct MailerQ to skip all the other plugins and continue with the normal work, or that MailerQ should start all over again with calling the plugins.
Be aware that you really must call one of these functions to give back the control to MailerQ. If your plugin fails to return control, the message will not be picked up again by MailerQ for further processing, but will stay in memory. This will eventually lead to MailerQ running out of resources.
With the first of the listed functions, MQ_continue(), your plugin instructs MailerQ to advance in the list of plugins, and to call the next plugin. Naturally, all next plugins have the same options as this plugin, and the subsequent plugins will work with the message as adapted by the current plugin. If no other plugins exist, MailerQ will move on to the next step in the process.
When the MQ_complete() function is called, MailerQ advances to the next step in the process. This means that any plugins in the list after the plugin will be skipped and MailerQ moves on without calling these plugins.
Lastly, your plugin can call MQ_retry() to instruct MailerQ to restart its plugin choosing process in the current step, effectively restarting the step from the first plugin. Beware though, since MailerQ will restart from the beginning of the list, all plugins prior to the current one will run again, possibly causing the altered message to be processed again.
Warning: You should take extra care when returning control of the message to MailerQ with the retry function, because it is possible to inadvertently create a loop. For most plugins, MQ_continue() is the best function to give back control to MailerQ.
MailerQ calls a plugin callback function. This function should return true if it wants to take over control from MailerQ (for example to do a non-blocking asynchronous operation). When it is ready and wants to hand over control to the next plugin or back to MailerQ, it should call MQ_continue(). In special circumstances you can call MQ_complete() or MQ_retry() instead.
To run MailerQ, you need a valid (free) license file. This file contains the list of IP addresses from which you are going to send out mail, and the features that should be enabled.
If you already have a license, but want to install a newer version of MailerQ, you need a new license file. This new license file can also be retrieved for free via the above hyperlink.
The MailerQ application has a special command line switch to get your valid license file. After you've installed MailerQ, and generated a license key on this website, run the following command to download the license:
sudo mailerq --fetch-license LICENSE_KEYYou have to replace LICENSE_KEY with your (trial) license key
A free trial license file is valid for a period of one month.
It is best to store your license file as "etc/mailerq/license.txt". If you store it in a different location, you have to update your config file.
If you have questions about your license, feel free to send an email to info@mailerq.com.
When you write your own MailerQ plugin, you must implement the mq_version() function. This function is called by MailerQ to find out if your plugin is compatible with the MailerQ binary.
Normally, this function can be very simple and just has to return the MQ_API_VERSION value. This is an integer value that increments with every new version of MailerQ.
#include <mailerq.h>
/**
* Function called by MailerQ to find out if the plugin is compatible
* with the MailerQ executable
*
* @return int API version number for which the plugin is compiled
*/
MQ_EXPORT int mq_version() {
return MQ_API_VERSION;
}
If the returned version number is not valid, MailerQ will report an error message and quit. If the function number is valid, MailerQ will continue to run, and will call your mq_initialize() function.
Below you will find a short list of our most frequently asked questions about our Mail Transfer Agent. For more information please see our documentation. If you cannot find what you are looking for, please contact support.
Before you download and install MailerQ, make sure you have a working RabbitMQ message broker (version 3.3.1+) . There are three ways to send email with MailerQ. However, posting messages directly to the AMQP message queue is the fastest way. Below you will find the 'technical' questions that are asked most frequent.
Yes, MailerQ supports IPv6. However, do note that not all receiving mail servers have support for IPv6. It is therefore recommended to use IPv4.
No, statistics, such as opens, clicks and more are handled outside of MailerQ. MailerQ focuses completely on the delivery process of your e-mails. All delivery related results are registered by MailerQ and returned to the RabbitMQ results queue. Here your own application can read and process these results.
There are many reasons why an email cannot be delivered (initially). MailerQ will automatically place these e-mails in the retry queue and send them again after a set amount of time. Each attempt will also add a new "results" property to the JSON so you will know exactly what happened during the delivery of the email.
After downloading and installing MailerQ, you will need a license to be able to use the software. This license holds a special 'key', the end date and the IP addresses you have specified as the ones you want to use.
You can find MailerQ licenses on the create License page. On this you can choose to create a Free License (which allows you to use one IP address, send 1000 messages a minute and expires after one month) or a Commercial License (unlimited messages, unlimited IP addresses, renewed montly or annually and professional support). Make sure you add your VAT number when purchasing a commercial license. After reading the terms and conditions press the 'create' button, pay for your license if applicable, and you can download your license.
On the license page of your personal profile you can renew your current license. Renewing a license will create the exact same license, with the same IP addresses, but with a different end date. Make sure you use the 'create new license' option when you want to add new IP addreses.
In the license page of your personal profile you can download any of your current and old licenses.
No. Once you have purchased your license we will not charge you anything extra for the services offered within that license.
MailerQ offers a setting to uses IP addresses that are not local to the server MailerQ is running on. We call these External MTAs.
A GET request returns all External MTAs currently configured. It returns a JSON array of MTAs, see the example below.
GET /v1/externalmtas HTTP/1.0
Authorization: Bearer ...Might result in
HTTP/1.0 200 Ok
Content-Type: application/json
...
[
{
"public_ip": "8.8.8.8",
"local_ip": "127.0.0.1",
"connect_port": "2525",
"protocol": "nat"
},
{
"public_ip": "8.8.4.4",
"local_ip": "192.168.1.1",
"connect_port": "2525",
"protocol": "nat"
}
]In this example, there are 2 External MTAs defined.
A POST request to allows you to add a new External MTA. For the request format, see the table below.
| Field | Required | Type | Description |
|---|---|---|---|
| public_ip | yes | string | The external IP address to use for sending |
| local_ip | yes | string | The local IP address to bind to |
| connect_ip | no | string | The IP to use for the external server |
| connect_port | yes | int | The port to use for the external server |
| protocol | yes | string | The protocal to use to connect to the external server ('nat' / 'socks' / 'http') |
For example, the request below will create the External MTA from the first example under GET with a json formatted body
POST /v1/externalmtas HTTP/1.0
Content-Type: application/json
Authorization: Bearer ...
{
"public_ip": "8.8.8.8",
"local_ip": "127.0.0.1",
"connect_port": "2525"
}and equivalently, with urlencoded body
POST /v1/externalmtas HTTP/1.0
Content-Type: application/x-www-form-urlencoded
Authorization: Bearer ...
public_ip=8.8.8.8&local_ip=127.0.0.1&connect_port=2525A DELETE request to allows you to remove an External MTA. This only needs the external IP address:
DELETE /v1/externalmtas HTTP/1.0
Authorization: Bearer ...
Content-Type: application/x-www-form-urlencoded
public_ip=8.8.8.8DELETE requests, like post requests, support both JSON and urlencoded formats.
You can run multiple MailerQ instances on a single server. But it does require some extra work.
If you run multiple instances on the same machine, these instances will still all load their configuration from the central /etc/mailerq/config.txt config file. This is no problem for most of the settings, but some specific settings need to different for each instance. For example, the different instances all need a seperate lock file, and have to use different port numbers to listen to.
If you want to run multiple instances at the same time, you still need to create a central /etc/mailerq/config.txt file with MailerQ's configuration. This file contain the settings that apply to all instances. But besides this central config file, you also need additional config files (for example /etc/mailerq/instance-1.txt, /etc/mailerq/instance-2.txt, et cetera) for the individual instances. In these config file you put the instance-specific settings.
If you start a MailerQ instance, you have to use the --config-file command
line option to specify the configuration file to use:
mailerq --config-file /etc/mailerq/instance-2.txtWith the above command you start a MailerQ process that loads both the normal /etc/mailerq/config.txt file, as well as the additional /etc/mailerq/instance-2.txt file.
When MailerQ starts, the settings are loaded in the following order:
1. Hardcoded default settings
2. Default config file (/etc/mailerq/config.txt)
3. Config file set in --config-file argument (optional)
4. Command line arguments (optional)To simplify the configuration files, you can use the default config file for all the shared settings and instance specific files with just the unique options.
There are a couple of settings that almost always must be unique if you run more than once instance:
The outbox queue must be unique per instance to manage which messages are sent by which MailerQ instance. Port number are limited resources and it is normally not possible to have multiple processes that all listen to the same port number. These settings have to be unique too:
rabbitmq-outbox: outbox1
smtp-ip: 10.0.0.1
www-ip: 10.0.0.1
www-host: mta001.example.comTo avoid corruption (you don't want that different processes write to the same log file at the same time) it is advised to use separate log files for the instances. Make sure to create the log directories and change ownership to the MailerQ user. Configuration for a MailerQ instance could look like this:
application-log: /var/log/mailerq/instance1/errors.log
send-log: /var/log/mailerq/instance1/send.log
download-log: /var/log/mailerq/instance1/download.log
received-log: /var/log/mailerq/instance1/received.logTo prevent a MailerQ instance from starting more than once, MailerQ stores its process ID (pid) in a lockfile. This must be unique for every instance:
lock: /tmp/mailerq-instance1.pidEach message that is accepted by MailerQ gets a unique message ID. To prevent that multiple instances assign exactly the same ID to a message, you can assign a "server-id" variable. This identifier guarantees that message ID's never conflict.
server-id: 1We recommend to enable the clustering when running multiple MailerQ instances. Because when changes are applied in one web interface, they are also immediately applied to all instances in a cluster. To enable clustering set the following options in the default configuration file:
cluster-address: amqp://user:password@rabbitmq.example.com/
cluster-exchange: clusterIf you like to use init scripts to start multiple MailerQ instances, you can copy the default init script:
$ sudo cp /etc/init.d/mailerq /etc/init.d/mailerq-instance1and alter the following settings inside the copied file (/etc/init.d/mailerq-instance1):
NAME="mailerq-instance1"
DESC="MailerQ daemon instance1"
PIDFILE="/tmp/mailerq-instance1.pid"
DAEMON="/usr/bin/mailerq --config-file /etc/mailerq/instance1.txt"Now you can start your instances from the terminal like:
$ sudo service mailerq-instance1 startor
$ sudo systemctl start mailerq-instance1To keep instances running you can use a process supervisor like Monit. With our previous example, a simple Monit script for MailerQ instance looks like:
check process mailerq with pidfile /tmp/mailerq-instance1.pid
group mail
start program = "/etc/init.d/mailerq-instance1 start"
stop program = "/etc/init.d/mailerq-instance1 stop"
if failed host 10.0.0.1 port 25 protocol smtp then restartAlternatively you can use cron to restart your MailerQ instances if they are not running. This can be set in the crontab for either the root or MailerQ user.
* * * * * kill -0 $(cat /tmp/mailerq-instance1.pid) || /usr/bin/mailerq --config-file /etc/mailerq/instance1.txt &
* * * * * kill -0 $(cat /tmp/mailerq-instance2.pid) || /usr/bin/mailerq --config-file /etc/mailerq/instance2.txt &An example of two configuration files for different instances:
/etc/mailerq/instance1.txt:
rabbitmq-outbox: outbox1
rabbitmq-inbox: outbox1
smtp-ip: 10.0.0.1
www-ip: 10.0.0.1
www-host: mta001.example.com
application-log: /var/log/mailerq/instance1-errors.log
send-log: /var/log/mailerq/instance1-attempts.log
download-log: /var/log/mailerq/instance1-downloads.log
received-log: /var/log/mailerq/instance1-received.log
lock: /tmp/mailerq-instance1.pid
server-id: 1/etc/mailerq/instance2.txt:
rabbitmq-outbox: outbox2
rabbitmq-inbox: outbox2
smtp-ip: 10.0.0.2
www-ip: 10.0.0.2
www-host: mta002.example.com
application-log: /var/log/mailerq/instance2-errors.log
send-log: /var/log/mailerq/instance2-attempts.log
download-log: /var/log/mailerq/instance2-downloads.log
received-log: /var/log/mailerq/instance2-received.log
lock: /tmp/mailerq-instance2.pid
server-id: 2MailerQ expects all messages that are loaded from the RabbitMQ outbox queue to be JSON encoded. This means that if you want to inject email directly into RabbitMQ, you must use JSON. In this JSON you should set the envelope ("MAIL FROM") address, the recipient ("RCPT TO") address and the full mime data to be sent.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "..."
}Strictly speaking, the "envelope" property is not even required, because the SMTP protocol also permits deliveries without an envelope address ("MAIL FROM:<>"). However, without an envelope you cannot receive bounces, delivery status notifications and/or out-of-office replies.
To make the examples a little easier to read, we often remove the mime data from the example JSON, and replace it with "...".
Besides the three properties demonstrated above, you can add all other kind of other properties to the JSON object to control the delivery of the mail. The following properties are recognized by MailerQ:
| envelope | envelope ("MAIL FROM") address |
| recipient | recipient ("RCPT TO") address |
| mime | full mime data to be sent |
| key | key to the mime data in external storage |
| keepmime | do not remove mime data after delivery |
| data | personalization data |
| ips | ip addresses to send the message from |
| pool | pool identifier to send the message from |
| nextattempt | properties for when the next attempt should be |
| maxdelivertime | time until which a delivery should be retried |
| maxattempts | max number of delivery attempts |
| retries | the delays between the delivery attempts |
| force | force delivery, even when errors occur or conversion is impossible |
| inlinecss | turn style blocks in html mails into inline style attributes |
| dkim | private keys to sign the mail |
| dsn | settings for delivery status notifications |
| queues | alternative rabbitmq queues for results |
| smarthost | smarthost settings |
| tags | the tags to add to the message |
| headers | add or change the mime headers |
| priority | the priority of the message |
| security | the security preference of the message |
The most simple JSON to send to RabbitMQ contains just an "envelope", "recipient" and "mime" property as we demonstrated above. The "mime" property should be a string value, holding a valid MIME object. This mime object holds the entire email, including all the headers and possible text and html versions, attachments, and so on:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "From: my-sender-address@my-domain.com\r\nTo: info@example.org\r\nSubject: ..."
}However, the "mime" property can also be a nested JSON object, holding all individual properties of the mime, which will then be turned into a valid mime string by MailerQ:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": {
"from": "my-sender-address@my-domain.com",
"to": "info@example.org",
"subject": "This is the subject line",
"text": "text version of the mail"
}
}The number of properties that are supported inside the nested "mime" property is pretty huge. It uses the very same algorithm as the responsiveemail.com web service to convert JSON objects into valid MIME data. To give you an idea of a possible valid JSON input, consider this:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": {
"from": "my-sender-address@my-domain.com",
"to": "info@example.org",
"subject": "Example subject",
"text": "This is the example message text",
"headers": {
"x-my-special-header": "special-value"
},
"content": {
"blocks": [ {
"type": "image",
"src": "http://www.example.com/logo.png"
}, {
"type": "button",
"label": "Click the button!",
"link": {
"url": "http://www.mailerq.com"
}
} ]
},
"attachments": [ {
"url": "http://www.example.com/a-special-file.pdf",
"name": "brochure.pdf"
} ]
}
}The number of options is huge. You can construct responsive emails using images, HTML text, social media, et cetera. MailerQ downloads all the resources and converts the JSON code into a valid MIME strings.
For more information about the supported JSON properties nested under the "mime" property, check the responsive email documentation.
MIME bodies can become large, especially if your emails contain attachments or embedded content. To prevent that such big messages have to be processed by RabbitMQ, MailerQ can be configured to use an external message store (for example: MongoDB). You can then publish much smaller JSON objects to RabbitMQ, and keep the full MIME messages in the external message store.
MailerQ waits with loading the message from message store until the SMTP connection has been set up, so no time and resources are wasted on fetching information that is not needed. The mime data is only loaded when it is really needed.
With an external message store, you can use the "key" property instead of the "mime" property. This property holds the key under which the mime data is stored in your external storage system.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"key": "message-store-key-where-data-can-be-found"
}Even when you have a message store you can of course still use the "mime" property and store the full mime in the JSON. This is often even the recommended way of doing things, because most mails get delivered at the very first attempt, and storing-and-fetching the mail to and from external storage is then only a waste of time and resources. The message store is in such a setup only used for messages that were delayed and that are published back to RabbitMQ. MailerQ takes care of removing the mime from the JSON, and stores it in the message store when delayed messages are published back to RabbitMQ.
You may be injecting different types of messages into MailerQ, for example messages belonging to your various customers or campaigns. To help you get a better overview, MailerQ allows you to tag your messages with one or more labels of your choosing. These tags will show up in the management console, allowing you to monitor and control all deliveries belonging to a tag. To specify the tags in the JSON you can add a "tags" property with an array containing all the labels:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"tags": ["Customer Name", "Example Campaign"]
...
}This is just an example, you are free to add any number of tags to your messages and interpret them as you wish. There are virtually no restrictions on the format of a tag; all characters including punctuation and international characters are accepted in the tag specification as long as the message is valid JSON.
When a message has been completely processed - either because it was successfully delivered, or the delivery failed - MailerQ publishes the delivery result to the results queue, where you can pick it up for further processing.
By default, MailerQ throws away the mime data to make room in the JSON object and in the message store. The result queues only hold the meta data for each sent message, but not the full mime data. If you do not want the message data to be removed, you can indicate this by adding the "keepmime" option:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"keepmime": true
}It is possible to include personalization data in the JSON. If you do this, MailerQ will treat the subject, html and text version of your email as templates, and replaces variables in it with the values loaded from the JSON.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "From: ...\r\nTo: ...\r\nSubject: Message from {$firstname}\r\n...",
"data": {
"firstname": "Emiel",
"lastname": "Bruijntjes",
"email": "emiel.bruijntjes@copernica.com"
}
}Personalization variables can currently only be used for the subject and the text and HTML versions of the mail. However, if you assign a nested object to the "mime" property using the responsive email specification, you can use personalization data in almost all fields.
For more information about using personalization, check our personalization documentation.
MailerQ can send out mail from all IP addresses configured on your server and will always try to make as many connections as it takes to send out your mailing. This can be useful to increase the delivery rate, because receivers often restrict deliveries per IP address.
It is however wise to limit the range of local IP addresses that a mailing can use, as you may not want to expose all your IPs to every campaign or group of mailings you send. The reputation of your IPs is important for getting your mailings delivered, so you may not want to risk that a single dubious campaign greylists all your IP addresses. (Especially if these messages are provided by third parties) Setting a predetermined range in the JSON can help prevent these sorts of problems.
To limit the range of IP addresses that a specific mailing can use, you may specify an array of "ips" as a property in the JSON body:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"ips": ["231.34.13.156", "231.34.13.158"]
}MailerQ will then only try to send the message from these IPs. Note that you can of course only use addresses that are actually bound to the host that MailerQ runs on. If the array provided is empty or if it only contains IP addresses that are not bound to your server, this will result in a failed delivery.
If no property "ips" is set in the JSON or if it is set to something other than a JSON array, MailerQ will fall back on a list of default IPs, which can be defined in the config with the property "smtp-defaultips":
smtp-defaultips: 1.2.3.4; 1.2.3.5; 1.2.3.6If this property is missing or left empty, MailerQ will instead opt to use ALL available IP addresses on the host server. However, we do recommend recommend that you set this property if you have many IPs configured on your server.
Outside of being able to set single IP addresses, a pool can also be specified. First, setup an IP Pool in the Management Console and assign some IPs to the pool. Then, to set the pool, you may simply set the identifier in the json:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"pool": "example-pool"
}MailerQ will then only use the IPs that are currently assigned to that pool. This way, if an IP performs poorly, it can easily be removed from the pool and MailerQ will no longer use it for the messages with the pool assigned, instead of having to rewrite every specific message to remove the IP where applicable.
Note that the pools are mutually exclusive with the IPs, that is, if the ips property is set in the JSON, the pool
property will be ignored, as a specific set of IPs is more specific than a pool setting.
Messages loaded by MailerQ are delivered right away. If you want to delay a delivery, you can add a "nextattempt" property to the JSON with the desired time of delivery for the next attempt.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"nextattempt": {
"time": "2019-01-10 00:00:00"
}
}In normal circumstances you want messages to be delivered right away, so you do not need this "nextattempt" property. However, under the hood, MailerQ uses this "nextattempt" property when a mail cannot be delivered because of greylisting. The message is then published back to the outbox queue with a "nextattempt" property set to a couple of minutes after the initial attempt.
This property is also set when Response Patterns cause a mail to be retried again after a specified interval.
When a message cannot be delivered immediately because of unresponsive receivers, greylisting or throttling, MailerQ publishes back the email to the outbox queue for later delivery. This can result in emails that are sent much later than the time that you first added them to the message queue.
If you do not want an email to be delivered after a certain time, all you need to do is add the "maxdelivertime" property. MailerQ will give up on delivering the mail after this time, and send a failure to the results queue.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"maxdelivertime": "2016-02-10 00:00:00",
"maxattempts": 3,
"retries": [ 600, 600, 1800, 3600 ]
}The "maxattempts" is an alternative way to control after how many attempts MailerQ should give up trying. The "maxattempts" setting limits the number of attempts MailerQ makes before it gives up. You normally should not use this setting because "maxdelivertime" is much more precise.
Timestamps in MailerQ are always in UTC. MailerQ is not very tolerant in parsing timestamps, so make sure that you use the right formatting (YYYY-MM-DD HH:MM:SS). If you do not specify an explicit max delivery time or max attempts setting in the JSON, and a default value has also not been set in the config file, MailerQ will attempt to deliver the mail within 24 hours after the mail was first picked up from the outbox.
The "retries" property can be added to specify exactly how long MailerQ should wait until the next delivery of the mail. This should be set to a JSON array holding the number of seconds between the attempts. In the above example, the first and second retry are started 10 minutes after the previous attempt. If the third attempt (which is the second retry) fails too, MailerQ will wait half an hour (1800 seconds). If that attempt fails too, all subsequent attempts will be sent with one hour intervals (3600 seconds).
When you send out HTML emails, you face the problem that not all email clients support stylesheets that are set in the header. Some email clients (especially web based clients) strip out the CSS code from the HTML header. This often messes up the layout and look of your messages. To overcome this, it is better not to set CSS settings in the header in the first place, but use "style=..." attributes in the HTML code.
MailerQ can do this automatically. If you set "inlinecss" property to true, MailerQ parses the HTML email, and converts the CSS code from the HTML header into inline "style=..." attributes in the HTML body.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"inlinecss": true
} You can include private DKIM Keys in the JSON to let MailerQ sign the mail.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"dkim": {
"domain": "example.com",
"selector": "x",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"expire" : "2017-01-01 00:00:00"
}
} It is also possible to include multiple keys in the JSON. The "dkim" property supports arrays:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"dkim": [ {
"domain": "example.com",
"selector": "x",
"key": "-----BEGIN RSA PRIVATE KEY-----\n....."
}, {
"domain": "example.com",
"selector": "y",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"expire" : "2017-01-01 00:00:00"
} ]
} Besides the private keys that you include in the JSON, MailerQ also keeps a set of private keys in its local database (and that can be edited using the management console). If there are multiple matching keys, they are all used for signing the mail.
By default, only regular headers are used for the signature: the "from" and "to" address, the "subject" and so on. If you want to include your own custom headers in the signature too, you can pass in an extra "headers" option:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"dkim": {
"domain": "example.com",
"selector": "x",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"headers": [ "x-custom-header", "x-other-header" ]
}
} The headers that you included in the "headers" option are also signed. This is especially useful if you want to include the "feedback-id" header in a signature. This "feedback-id" header is required to be signed for Google/Gmail feedback loops.
If you want to receive reports from remote servers whenever a DKIM signature fails to verify you can add a flag to the DKIM object to indicate this:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"dkim": {
"domain": "example.com",
"selector": "x",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"report": true
}
}Note that you will have to update your DNS records with the options defined in RFC 6651 for the remote server to know where it should send their reports.
A DKIM key can also be used for creation of ARC signatures. This is enabled by setting the "protocols" option. This option should be an array of the protocols you want to use (DKIM and/or ARC):
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"dkim": [ {
"domain": "example.com",
"selector": "x",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"protocols" : ["dkim", "arc"]
}, {
"domain": "example.com",
"selector": "y",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"protocols" : ["arc"]
} ]
} The first key is used for both DKIM and ARC signing, whereas the second key is only used for ARC signing.
A DKIM can be signed using different forms of canonicalization. Default is relaxed/simple.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"dkim": [ {
"domain": "example.com",
"selector": "x",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"canonicalization" : "simple/simple"
}, {
"domain": "example.com",
"selector": "y",
"key": "-----BEGIN RSA PRIVATE KEY-----\n.....",
"canonicalization": "relaxed/relaxed"
} ]
} In the example, the first key is signed with simple/simple canonicalization, and the second
key is signed with relaxed/relaxed canonicalization.
The "dsn" property can be added to control whether MailerQ should send back an email to the envelope address in case of a failed delivery.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"dsn": {
"notify": "FAILURE",
"ret": "HDRS",
"orcpt": "info@example.org",
"envid": "my-identifier",
}
}The above JSON contains a DSN setting that says that a delivery status notification should be sent back to the original envelope address in case of a failure (this is what the "notify" setting says). The notification should include the headers of the original mail (ret=HDRS). The notification should specify that the original recipient was "info@example.com", and the unique envelope identifier was "my-identifier".
For more information about delivery notifications, see the Delivery Status Notification documentation.
After MailerQ has processed an email, it publishes the mail to one or more result queues. The default queues to use for this are configured in the global configuration file, but you can set other queue names to tell MailerQ to use other result queues.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"queues": {
"results": "name-of-results-queue",
"failure": "name-of-failure-queue",
"success": "name-of-success-queue",
"retry": "name-of-retry-queue",
"dsn": "name-of-dsn-queue"
}
}All properties inside the "queues" object are optional. If you leave an option out, MailerQ will use the default queue from the config file. If you include a queue but set it to a null value, the queue will not be used. Thus, if you for example want to process only the errors for a certain e-mail, you can only set the "failure" queue, and set all other queues to null. When the delivery succeeds, MailerQ will silently discard the mail, without adding it to any result queue:
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"queues": {
"results": null,
"failure": "name-of-failure-queue",
"success": null
}
}There is an important difference between not mentioning a queue in the "queues" object, and setting it to null. In the above example, the "dsn" and "retry" queues are missing from the "queues" object. This means that for these two queues the setting from the config file will be used (set with the "rabbitmq-dsn" and "rabbitmq-retry" options). For the "results" and "success" queues a null parameter was used, so that the settings from the config file are ignored, and no messages are ever published to the results or success queues.
If you do not want to send the message right to the recipient right away, but to an alternative SMTP server on the internet, you can add a "smarthost" option.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"smarthost": {
"hostname": "mail.smtpeter.com",
"port": 25,
"username": "my-username",
"password": "my-password"
}
}The above message will not be sent to "example.org", but to "mail.smtpeter.com" instead.
MailerQ can be used to alter the MIME headers. To do so, you can add a "headers" option.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "From: ex@ex.com\r\nTo: info@ex.com\r\nDummy: xyz\r\nSubject: ...",
"headers": {
"remove" : ["dummy", "bcc"],
"prepend" : {
"cc" : "cc@ex.com",
"x-example" : ["example 1", "example 2"]
},
"append" : ...,
"replace" : {
"from" : "xe@xe.com"
}
"update" : ...,
}
}The "remove" property can be a string or an array of the header fields that have to be removed from the mime. Using the "prepend" option one can prepend headers to the mime. Multiple headers with the same key can be prepended by supplying an array of values. The "append" (or "add") option holds the headers that have to appended (same syntax as prepend). The headers described in "replace" will replace the first occurrence and remove the other headers with the same key. If no existing header is found, it will simply be appended. The "update" option works similar, but it non-existing headers will not be appended.
To give the message a priority, the priority field should be used. This setting will only have
an effect if the rabbitmq-maxpriority is set in the config file, since the queues will then be
constructed as priority queues.
Since MailerQ 5.12, emails themselves can now also be given a security value. There are 4 different values for this field.
| optional | the message will make use of whatever is available, and will not specifically use STARTTLS |
| opportunistic | the message will use STARTTLS if it is available, but otherwise sends over an unsecured connection |
| required | a secure connection is required for the message to be sent |
| verified | the connection has to be secure, and the certificate and cipher have to match the email domain |
For example, the following email will require an SSL connection to be set up to the remote server. Do keep in mind that this is only for the first hop; as there is no standard for setting this property on emails, if the message is transferred from the other server to its final destination, there is no way to enforce it being done over a secure connection.
To have better control over your message queue, you can add additional properties in the JSON. These properties are ignored by MailerQ, but they will end up in the result queue, and allow you to link result data with the original mail.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"custom-property-name": "debug data",
"mime": "..."
}MailerQ has lots of configuration options. Each option can be supplied in three different ways:
/etc/mailerq/config.txt.When an environment variable is set, it overrides the value specified in the configuration file. Similarly, when a command-line option is provided, it overrides both the configuration file as well as the environment variable.
Overridability of keys follows a simple pattern. If you want to override such a
key via the command-line, add two dashes in front of the key name. If you want
to override the key via an environment variable, uppercase the key, prepend the
string MAILERQ_ to it, and replace dashes with underscores.
As an example, the configuration key rabbitmq-address can also be given on
the command-line by running
mailerq --rabbitmq-address=amqp://rabbit.example.comMailerQ will then use rabbit.example.com for its RabbitMQ server, instead of
the value that's given in /etc/mailerq/config.txt. Similarly, you can provide
an environment variable:
MAILERQ_RABBITMQ_ADDRESS=amqp://rabbit.example.com mailerqThe above will cause MailerQ to use rabbit.example.com. If you supply both,
then the command-line override "wins".
In the rest of this documentation, we only denote the name of the configuration key as specified in the configuration file.
The documentation for the various options is split up in different sections. Please select the section that you're interested in or find the desired setting in the A-Z settings overview.
The config file is not the only place where MailerQ gets its configuration from. Most of the settings relevant for deliveries are not kept in the config file, but are loaded from a relational database. The web based management console allows you to read or change these settings, and you can of course also access the database directly to read or update these settings.
MailerQ offers IP Pools for easy management of your sending IP addresses.
A GET request to pools returns all pools. It returns a JSON array of pools, see the example below.
GET /v1/pools HTTP/1.0
Authorization: Bearer ...Might result in
HTTP/1.0 200 Ok
Content-Type: application/json
...
[
{
"name": "sharedpool1"
},
{
"name": "dedicated",
"description": "Dedicated Addresses"
}
]In this example, there are 2 pools defined.
To get the actual IP addresses in a pool, you can do a GET request to poolips?name=NAME. The example below returns the IPs in pool "sharedpool1" from the example above.
GET /v1/poolips?name=sharedpool1 HTTP/1.0
Authorization: Bearer ...This will return the list of IP addresses currently in the pool called NAME:
HTTP/1.0 200 Ok
Content-Type: application/json
...
[
{
"ip": "127.0.0.1",
"name": "sharedpool1"
},
{
"ip": "192.168.1.1",
"name": "sharedpool1"
}
]A POST request to pools allows you to create a new IP Pool. For the request format, see the table below.
| Field | Required | Type | Description |
|---|---|---|---|
| name | yes | string | The name of the new pool |
| description | no | string | Description of the pool, for human use. |
For example, the request below will create the pool from the second example under GET with a json formatted body
POST /v1/pools HTTP/1.0
Content-Type: application/json
Authorization: Bearer ...
{
"name": "dedicated",
"description": "Dedicated Addresses"
}and equivalently, with urlencoded body
POST /v1/pools HTTP/1.0
Content-Type: application/x-www-form-urlencoded
Authorization: Bearer ...
name=dedicated&description=Dedicated+AddressesTo add an IP address to an existing pool, you can do a POST request to poolips. This needs the name of the pool and the address to be added:
| Field | Required | Type | Description |
|---|---|---|---|
| ip | yes | string | The name of the existing pool |
| name | yes | string | The address to be added |
POST /v1/poolips HTTP/1.0
Content-Type: application/json
Authorization: Bearer ...
{
"ip": "127.0.0.1",
"name": "sharedpool1"
},Naturally, POST requests to poolips can also be done with urlencoded data.
A DELETE request to pools allows you to remove a pool. This only needs the name of the pool to be removed:
DELETE /v1/pools HTTP/1.0
Authorization: Bearer ...
Content-Type: application/x-www-form-urlencoded
name=sharedpool1This will also remove all addresses associated with that pool.
To remove a specific address from a pool, you can do a call to poolips with the name of the pool and the address you want to remove:
DELETE /v1/poolips HTTP/1.0
Authorization: Bearer ...
Content-Type: application/x-www-form-urlencoded
name=sharedpool1&ip=127.0.0.1DELETE requests, like post requests, support both JSON and urlencoded formats.
MailerQ runs on Linux. We distribute the software in binary form for Debian/Ubuntu based systems and for Red Hat based systems. Please drop us a message if you need a version for a different type of system.
Most libraries that are needed by MailerQ are statically linked. This means that most of the required libraries are embedded into the binary code, so MailerQ runs straight out of the box with no need for you to fix dependencies and install other software. However, when MailerQ starts, it does do a scan of your system to detect which libraries are available. If MailerQ happens to find one or more of the following libraries, it does load them to use specific features from these libs:
| Tables | Description |
|---|---|
| libopenssl | Used for TLS encryption, license checking and base64 encoding |
| libz | Used for compression algorithms |
| libmagic | Used to detect the mime-type of files |
| libuuid1 | Used for generating globally unique identifiers |
| libidn | Used for parsing international domain names |
| libxml2 | Used for parsing and modifying XML/HTML code |
| libimagemagick | Used to find out the dimensions of images |
| libmysqlclient | Used to connect to a mysql/mariadb database |
| libmariadbclient | Used to connect to a mysql/mariadb database |
| libpq | Used to to connect to a Postgresql database |
| libsqlite3 | Used to process sqlite3 database files |
| libmongo-c-driver | Used to connect to a MongoDB NoSQL server |
| libcouchbase | Used to connect to a Couchbase server |
From the above list, the openssl, libz and libidn libraries are required. All other libraries are optional and MailerQ can run without them. If a library is missing, MailerQ will either fall back on its own implementation, or will run without the features from the specific library.
Most of the MailerQ runtime settings are stored in a relational database. This means that you must have at least one of the mysql, mariadb, postgresql or sqlite3 libraries installed on your system.
Sqlite3 is a local solution. Mysql or Mariadb can run on a different servers, but then you need libmysqlclient on the mailerq box. Postgresql 9.5+ which can be on a different server too, but then you need libpq >= 9.5 to use it.
MailerQ can personalize email messages on the fly. If you add an extra "data" property to the input JSON, MailerQ treats the subject line and the text and HTML versions of your mime messages as templates, and replaces all variables in them with the values from the JSON "data" property. You can use these values in the "from" and "to" address too.
{
"recipient": "john@example.com",
"mime": "....",
"data": {
"ourname": "The MailerQ test team",
"name": "John Doe",
"age": 33,
"job": "programmer",
"children": [
"Peter", "Angela", "Brandon"
]
}
}If you use the above JSON data for your mail, you can use these variables inside the "from" and "to" address, the subject line, and inside the text and HTML versions of your email.
From: {$ourname} <info@example.org>
To: {$name} <{$recipient}>
Subject: Hello {$name}!
Content-Type: text/plain
Hi {$name},
Your age is {$age}, and your job is {$job}.
Cheers,
{$ourname}If you had used the above MIME as input, MailerQ would replace the variables in the "from", and "to" address, the subject line and the text and HTML versions. For ease of use the "envelope" and "recipient" are already extracted from the mail for you. In case you don't want to personalize at all because your body has to contain {$name} for example, you can disable personalizing by omitting the data property entirely.
Read our programming page if you want to know more about how to use the personalization capabilities.
The HttpCaller is a Linux application for making outgoing HTTP calls. It reads instructions from a RabbitMQ message queue, and turns them into HTTP POST calls. We wrote this application for MailerQ users who asked for a solution to send the results of the deliveries to a HTTP endpoint. It is however a generic application: it can be used for all sort of calls to HTTP endpoints.
The application is stored in our APT repository. If you have already enabled our APT repository, you can install it with this command:
sudo apt install mailerq-httpcallerA typical use case is to route the MailerQ delivery results to this application. However, MailerQ output is not compatible with HttpCaller input. The output of MailerQ must thus first be converted. We have a special application for this: mailerq-amqphttp. There is also a simple installation script that runs the entire stack of the httpcaller and the converter: mailerq-webhook.
You typically run this program from the command line. It reads all its input from a config file stored in /etc/copernica/httpcaller.txt. However, you can also pass command line arguments, for example:
httpcaller --rabbitmq-address amqp://user:password/hostname/vhostAll options can be supplied in the system wide config file, via environment variables, and as command line options. The config file is stored in /etc/copernica/httpcaller.txt. The following options are all equivalent:
All other configuration settings can be converted in a similar manner.
HttpCaller reads all instructions from a RabbitMQ message queue, and writes the results back to RabbitMQ. The following options can be used to configure this:
HttpCaller needs access to a RabbitMQ server via the AMQP address to consume and publish messages. But because the AMQP lacks the ability to find out which queues are active you must also specify the REST endpoint of your RabbitMQ server.
All messages are read from the configured queue, and the results (whether the HTTP POST call was a success or not) are published back to an exchange using routing keys "delayed", "failure" or "success". If you do not like the default routing keys, you can override them with other values.
If there is a big queue with outgoing POST calls for one specific domain, HttpCaller temporarily creates an overflow queue and parks all those calls in that queue. By doing so, it can still process other messages from the same queue for other domains. This prevents that lag for one domain limits the rate for other domains. To disable this feature, you can set the "overflow-queues" option to false.
HttpCaller can send out many HTTP POST calls at the same time. The limits are configurable:
The host connections limit can be overridden on a per-call bases.
Outgoing HTTP connections can be signed to allow receivers to verify that the call came from the expected source. There was a proposol for a standardized way to add such signatures to HTTP calls, and the httpcaller implements this protocol. If you only use httpcaller to make calls to your own servers over an internal network this feature might not be necessary, but for calls that go over the internet it can be helpful to avoid spoofing.
If you do not set these options, the HTTP calls will not be signed. The signing key can also be set per call (see below).
By default, the httpcaller uses the default DNS system resolver based on the settings from /etc/resolv.conf. But if you want to use a different DNS server, the following settings can be helpful:
In normal circumstances there is no need to change any of the DNS settings.
To prevent that httpcaller makes calls to forbidden webservers, you can install blacklists and/or whitelists.
By default, httpcaller is configured with the following blacklist: "192.168.0.0/16,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12". This means that all calls to the internal network are blocked, and that the application can only be used for calls to internet sites. If you use httpcaller for posting data to an internal endpoint, this must be overridden in the config file.
To be documented
The messages consumed from RabbitMQ must be JSON encoded, and must have a (minimal) form similar to this:
{
"url": "http://www.example.com",
"data": {
"field1": "value1",
"field2": "value2"
}
}The above instruction will be turned into a HTTP POST call to http://www.example.com with the supplied data as message body (converted to a "application/x-www-form-urlencoded" encoded message body). Alternatively, the data can also be supplied as string. But then you also pass a content-type as that cannot be auto-detected:
{
"url": "http://www.example.com",
"data": "textual post data",
"content-type": "text/plain"
}If you want to pass additional headers with the POST call, you can add that to the JSON as well:
{
"url": "http://www.example.com",
"data": "textual post data",
"content-type": "text/plain",
"headers": {
"x-my-extra-header": "my-value"
}
}The config file can be used to restrict the number of outgoing HTTP connections, but this can be overridden on a per-message basis:
{
"url": "http://www.example.com",
"data": "textual post data",
"content-type": "text/plain",
"connections": 10
}If you want to sign outgoing requests (see above), you can also override the settings from the config file:
{
"url": "http://www.example.com",
"data": "textual post data",
"content-type": "text/plain",
"sign": {
"id": "the key id",
"key": "private key in PEM format"
}
}HttpCallers posts the results of all calls back to RabbitMQ where you can inspect them. For each result, the original message is posted with an extra "results" array, holding one object for each attempt:
{
"url": "http://www.example.com",
"data": "textual post data",
"content-type": "text/plain",
"results": [{
"ip": "1.2.3.4",
"port": 80,
"code": 200,
"time": 1692108936,
"duration": 0.210,
"error": "optional error message",
"header": "full HTTP response header"
}]
}MailerQ can be accessed by REST API, and delivery flow can be altered. In this way, it allows you to stop the sending of certain emails, intercept emails, or inject emails directly into MailerQ with only a HTTP client.
Authentication to the API can be done using tokens that can be inserted into the database directly, or created in the web interface. To create a token, go to the management console Settings > REST API. Either copy an already existing token, or generate a new token with the button in the upper right corner.
To authenticate to the API, an Authorization header should be added to each request. For example, Authorization: Bearer <token> would be the correct way to authenticate to the API. Currently, the authentication tokens give full access to the complete API.
The following endpoints are available in the current version of MailerQ.
If you want to change the recipient address that is associated with a MQ_Message, you can use this function. This function allows you to store a new recipient address in a message.
/**
* Update the recipient address
*
* @param message the message to set the recipient address in
* @param recipient actual recipient address
* @param size length of the recipient string
*/
void MQ_setRecipient(MQ_Message *message, const char *recipient, size_t size);This function has a different behavior if you call it on a message that is being received and on a message that is being sent. If you call it on a message that is being received, for example inside your mq_smtp_in_message() function, it is simply an alias for setting the "recipient" property in the JSON. Changing the recipient address by calling this function, will also change the JSON that is going to be published to RabbitMQ.
For outgoing messages however, changing the recipient with this function will not modify the JSON data. If your plugin calls this function before the "RCPT TO" instruction is sent, the newly set recipient address is going to be used in the SMTP protocol, but when the message is published back to RabbitMQ (for example to the result queue, or back to the outbox queue for a retry) it will still have the original recipient address in the JSON. If you also want to modify the recipient address in the JSON data, you should use the MQ_json() function as well.
When an email is received by MailerQ, the mail is turned into a JSON object and published to the inbox queue. This happens to messages that are received on the SMTP port, but also to messages that come in via the spool directory or that were injected via the command line interface: every message is turned into a JSON object and published to the inbox queue.
The JSON messages that are published by MailerQ follow exactly the JSON specification for outgoing messages. It is therefore possible to let MailerQ publish these incoming messages directly to the outbox queue to deliver them to the final recipient.
MailerQ also adds some extra properties to the JSON of incoming messages that are not needed for the delivery of the email, but that are useful in case you write scripts or programs that process incoming messages. The following properties can be found on incoming messages:
| hostname | server name that received the message |
| received | time when the mail was received |
| message-id | unique message id generated for the mail (only used when received via smtp) |
| connection | connection info (only used when received via smtp) |
| spool | spool info (only used when received from spool dir) |
| cli | command info (only used when received from cli) |
| checks | array with the results of the SPF, DKIM and DMARC checks |
The above properties are never used by MailerQ when the email is sent, but they can be useful to recognize incoming messages.
When a connection comes in over an SMTP connection, MailerQ reports a unique internal identifier to the sender that can be used to track the email.
250-Ok, your message has been queued using the following identifiers:
250 gdsfu232 for info@example.comEvery recipient gets a unique identifier. This unique identifier is stored in the "message-id" property.
{
"message-id": "gdsfu232",
"received": "2016-03-22 17:23:12",
"hostname": "sender1.mailerq.com",
"envelope": "whatever@example.com",
"recipient": "info@example.com",
"mime": "..."
}The "received" property holds the time when the email was received, while "hostname" holds the name of the server that received the message. These two properties are included for all injection mechanisms: SMTP, CLI, and spool.
If the mail came in via an SMTP connection, the generated JSON object holds properties about the TCP connection.
{
"message-id": "gdsfu232",
"received": "2016-03-22 17:23:12",
"hostname": "sender1.mailerq.com",
"envelope": "whatever@example.com",
"recipient": "info@example.com",
"mime": "...",
"connection": {
"local-ip": "1.2.3.4",
"local-port": 25,
"remote-ip": "5.6.7.8",
"remote-port": 25324,
"secure": true,
"user": "mailerq"
}
}If MailerQ runs behind a HAProxy server, the connection data is extracted from the PROXY header that is sent by the HAProxy server.
If the TCP connection is secure and some sort of SMTP authentication mechanism was used, the "user" property holds the username of the user who submitted the message.
For mails injected to the spool dir, the JSON holds properties that identify the file from which the message was loaded.
{
"received": "2016-03-22 17:23:12",
"hostname": "sender1.mailerq.com",
"envelope": "whatever@example.com",
"recipient": "info@example.com",
"mime": "...",
"spool": {
"directory": "/path/to/directory",
"file": "filename.mime",
"user": "owner",
"size": 23299
}
}The properties speak for themselves: the spool directory path, name, owner and size of the injected file.
Message injected via MailerQ as command line utility hold information about the started program:
{
"received": "2016-03-22 17:23:12",
"hostname": "sender1.mailerq.com",
"envelope": "whatever@example.com",
"recipient": "info@example.com",
"mime": "...",
"cli": {
"command": "/usr/bin/mailerq",
"arguments": [ "--extract-recipients", "--ignore-dot" ],
"user": "john"
}
}In the above example, the email was injected using MailerQ as command line utility. The program was started by user "john", and two options were passed: "--extract-recipients" and "--ignore-dot".
Message injected via MailerQ REST API hold information about who connected
and the name of the authentication token in the http property.
{
"envelope": "whatever@example.com",
"recipient": "info@example.com",
"mime": "...",
"http": {
"authentication": "rest token name",
"remote-ip": "1.2.3.4",
"remote-port": 12345
}
}If you enable checks for incoming messages, MailerQ runs SPF, DKIM and DMARC checks on each incoming message. The result of these checks are added to the JSON.
{
"message-id": "gdsfu232",
"received": "2016-03-22 17:23:12",
"hostname": "sender1.mailerq.com",
"envelope": "whatever@example.com",
"recipient": "info@example.com",
"mime": "...",
"connection": {
"local-ip": "1.2.3.4",
"local-port": 25,
"remote-ip": "5.6.7.8",
"remote-port": 25324,
"secure": true,
"user": "mailerq"
},
"checks": [ {
"spf": "pass",
"smtp.mailfrom": "whatever@example.com"
}, {
"dkim": "pass",
"domain": "example.com",
"selector": "dkim",
"header.i": "fsjfksjfslkdf",
"header.b": "sodfidjsfdsjfsfjs"
}, {
"dmarc": "pass",
"header.from": "whatever@example.com"
} ]
}The properties for each test depend on the type of test, and hold the input parameters that were used to execute the test.
The checks are only executed for messages that are injected via SMTP, and only if checks are enabled in the config file using the "smtp-check" variable.
MailerQ calls this plugin function for every message that is received on the SMTP port. If you want to write a plugin that processes or modifies such messages, you can do so by implementing this function.
Your plugin may modify the JSON data in the MQ_Message. After all the plugins are ready with processing (and optionally modifying) the message, MailerQ will publish the JSON data to the RabbitMQ inbox queue.
Note that the SMTP protocol allows one to use multiple "RCPT TO" instructions to send a single MIME message to multiple addressees. If this is being used, the mq_smtp_in_message() function will also be called multiple times, once for every addressee.
bool mq_smtp_in_message(MQ_Context *context, MQ_Connection *connection, MQ_Message *message);
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
MailerQ offers Suppression List functionality to prevent unwanted recipients from being tried and protect your reputation.
A GET request returns all currently suppressed addresses. It returns a JSON array of addresses, see the example below.
GET /v1/suppressions HTTP/1.0
Authorization: Bearer ...Might result in
HTTP/1.0 200 Ok
Content-Type: application/json
...
[
{
"type": "address",
"value": "test@example.net",
"code": 551,
"extended": "5.1.1",
"description": "Spamtrap"
},
{
"type": "domain",
"value": "example.com",
"code": 550
}
]In this example, messages to test@example.net and example.com are failed with an error code 551 and 550 respectively,
with the former also having the extended error code 5.1.1 and error description Spamtrap attached to the result.
A POST request allows you to create a new Suppression List entry. For the request format, see the table below.
| Field | Required | Type | Description |
|---|---|---|---|
| value | yes | string | The domain or address to be suppressed |
| type | yes | string("address", "domain") | Whether the given value is a domain or a full address |
| code | yes | string | The error code given to suppressed messages. |
| extended | no | string | Extended SMTP error code, e.g. 5.7.1. |
| description | no | string | Description to put in the message result. |
For example, the request below will install the suppression from the first example under GET with a json formatted body
POST /v1/suppressions HTTP/1.0
Content-Type: application/json
Authorization: Bearer ...
{
"type": "address",
"value": "test@example.net",
"code": 551,
"extended": "5.1.1",
"description": "Spamtrap"
}and equivalently, with urlencoded body
POST /v1/suppressions HTTP/1.0
Content-Type: application/x-www-form-urlencoded
Authorization: Bearer ...
type=address&value=test%40example.net&code=551&extended=5.5.1&description=SpamtrapA DELETE request allows you to remove a suppression entry. The suppression is uniquely identified by the value (either a domain or a full address)
DELETE /v1/suppessions HTTP/1.0
Authorization: Bearer ...
Content-Type: application/x-www-form-urlencoded
value=example.comDELETE requests, like post requests, support both JSON and urlencoded formats.
MailerQ uses a relational database to store all sorts of configuration data and delivery settings. The data stored in the database includes the delivery throttles and DKIM Keys. The database is read and updated by MailerQ, but you are free to write your own scripts and programs that also modify this data.
MailerQ supports multiple database platforms: MySQL(5.5+), MariaDB(5.5+), PostgreSQL(9.1+) and SQLite3. The SQLite database is by far the simplest to set up because it does not require a server process to run. All you need is the "sqlite3" library to be installed on your system. Although it is the simplest to set up, we advise you to not use an SQLite database in production, because it makes scaling up MailerQ a lot more difficult.
The other database systems, MySQL, MariaDB and PostgreSQL, take a little more effort, but they are not too difficult to configure either. This is the most future-proof, since it can be easily scaled or shared between instances. If you already have such a database server running, you only have to create a new database and put the address, login and password to that database in MailerQ's configuration file. MailerQ will do the rest: the tables will be created and filled with the settings that you enter via the management console.
Only one variable has to be set in the config file to connect to a database: the "database-address" variable:
database-address: sqlite:///path/to/database/file.sql
database-address: mysql://user:password@hostname/databasename
database-address: postgresql://user:password@hostname/databasenameSQLite is the simplest database to set up, just specify the path to a file on the MailerQ server (this file does not even have to exist). However, an SQLite database can not be used if you want to access the same database from multiple machines.
If you do use SQLite, note that you need three (!) slashes in the address: the "sqlite://" prefix, followed by the "/path/to/database". If you use a MySQL, MariaDB or PostgreSQL database, make sure that the database exists, and that MailerQ not only gets enough privileges to read and write from and to the database, but also to create and modify tables.
On startup, MailerQ automatically checks if the tables in your database exist, and whether the database structure is correct. Missing, broken or incomplete tables are reported.
From a performance perspective it does not really matter which database engine you choose to use. MailerQ periodically copies all settings from the database to main memory, and uses this in-memory cache for lookups. No realtime queries are executed, and no connections are kept open to the database in between these reloads. The speed of the database does therefore not have to be a factor in choosing the most appropriate engine. It is better to choose a database that you feel most comfortable with. Do you already use MySQL databases? Then it is best to use it for MailerQ too. Do you run a single MailerQ instance, and does data not have to be shared amongst multiple MailerQ instances? Then the SQLite database is sufficient.
MailerQ makes use of client libraries on your system to connect to the database. To connect to a MySQL or MariaDB database, you must make sure that either libmysqlclient or libmariadbclient is installed on your system. For PostgreSQL connections libpq has to be installed, and libsqlite3 is needed for SQLite3 databases. If these libraries are not available on the system, it is not possible to connect to the database.
As we mentioned, the database is not used for real time queries. Instead, all data is periodically copied into main memory. This ensures that the speed of the database is never a limiting factor in your mail deliveries, or that a slow query slows down your email deliveries.
MailerQ normally reloads the database every 10 minutes. However, this is a configurable interval. In the config file you can set the "database-ttl" variable to any possible interval you like. This variable should be set to the number of seconds that the data loaded from the database is kept in main memory. The default value is 600, which is 10 minutes.
database-ttl: 60If the database-ttl is set to 60 seconds, like we did above, the data is reloaded every minute.
When MailerQ starts, it normally first connects to the database and checks whether
all tables are in a valid state. If this test fails, MailerQ does not start and
reports an error instead. With the --repair-database command line switch,
you can instruct MailerQ to repair all broken tables. Tables and columns that
are missing will be created, and obsolete columns are removed,
$ mailerq --repair-databaseThis option tells MailerQ to create and alter missing or broken tables. The only time when you normally do this, is when you install MailerQ for the first time, or when you are upgrading from a previous version.
If you want to enforce that all tables in the database are dropped and replaced by brand new empty tables, you can start MailerQ with the "--purge-database" command line option:
$ mailerq --purge-databaseThis option tells MailerQ not to check and repair tables, but to drop them all and create new ones.
You may want MailerQ to skip the database consistency check on startup. For example if you have manually modified the tables and added a couple of extra columns. To skip the validation on startup, you can set the config file variables 'database-validate' to false:
database-validate: falseIt is possible to run multiple MailerQ instances that all connect to the same database. If you do this, we recommend setting up a cluster of MailerQ instances. The different MailerQ instances then notify each other every time the settings in the database are updated, so that each instance can update its cache.
MailerQ has a powerful web based MTA management console. This console gives you access to the database using simple web forms. During normal operations, it is therefore not necessary to run any queries on the database yourself. But if you do like to access the data, you are free to do so.
All tables in the database have a very straightforward structure. Because MailerQ is compatible with many different database systems, we do not use vendor specific SQL features and have kept all tables as simple as possible. The tables use well known data types, and no foreign keys or constraints. Booleans are stored as integers and "NULL" and "0" values have the same semantics. The value "0" is often used to set something to "unlimited", while "-1" values are used to fall back on the default setting.
All data is stored in the tables in UTF-8 encoding. If you use international domain names and/or international email addresses, you should store them in UTF-8 format in the database.
The following tables are created:
| Tables | Description |
|---|---|
| capacity | all delivery capacities, eg max number of connections |
| capacity_rules | capacity rules per domain / MTA IP |
| dkim_keys | DKIM Keys to sign outgoing mails |
| dkim_patterns | rules that decide what DKIM Keys to use |
| dns | overridden dns lookups |
| errors | Forced Errors |
| flood_responses | alternative delivery limits to use when certain responses come in |
| ips | all MailerQ's IP addresses |
| locals | Local Email Addresses |
| mtanames | alternative "EHLO" names |
| paused | deliveries on pause |
| response_actions | response action, eg send mail from a different IP |
| response_patterns | patterns of responses that has to match |
| response_patterns_actions | what kind of response actions must be executed |
| rewrite_actions | actions to perform when a message is loaded eg send from a different IP |
| rewrite_triggers | triggers to fire rewrite actions |
Most tables are pretty self-explanatory, and every table can be filled by using the management console. We'll give a short explanation of each table here.
All delivery capacities are stored in this "capacity" table. It for example holds the max number of connections that may be set up to a certain domain, or the max number of messages to send within a minute.
The "localip" and "domain" columns in this table can both be empty. If they are empty, the setting applies to all local IPs and/or all domains. Only the most specific record is used. Thus, if a mail is sent from IP 1.2.3.4 to domain example.com, the record with the same localip and same domain is used. If no such record exists, MailerQ looks for a record that only matches the domain or the IP (and for which the other field is empty). If that record also does not exist, the record for which both the domain and the IP are empty is used.
The best way to get an understanding of the column in this table is to play around with the management console, and see what kind of data ends up in this table.
The "flood_responses" table has a similar layout to the capacity table, but with a "pattern" and a "type" column instead of an IP address and a domain. For each failed delivery, the answer message from the remote server is checked agains the patterns in this table. If there is a match, MailerQ will start using the delivery capacity set in this table.
For example, if you insert a record in this table with pattern "flood detected" and type "substr", and MailerQ receives an answer from a remote server with a message like "message rejected, flood detected from your IP address", MailerQ automatically switches the send capacity to the associated capacity.
Once again, the best way to get an idea of the meaning of the columns is to just experiment with the management console, and then look at the data in this table.
On startup, MailerQ detects all IP addresses that are linked to the server, and stores these IP addresses in a database table. MailerQ does not use this table for queries or for anything, it is just created so that other programs or scripts can find out which IP addresses are available for sending out emails.
All DKIM private keys that are used for signing outgoing mails are stored in the "dkim_keys" table. Only the domain, the selector and the expire time for the signature are stored; other properties are retrieved from DNS.
To decide what keys should be used for outgoing emails, there is a separate "dkim_patterns" table. The from address of every outgoing mail is compared to the records in this table, and whenever there is a match, the associated key is used for a DKIM signature.
The pattern type can be "regex", "fnmatch", "substr" or "exact", and decides whether the "sign" column holds a regular expression, or some other kind of pattern.
Incoming mails are compared with the addresses stored in the local email address table. When there is a match, the mail is not sent to the regular inbox RabbitMQ message queue, but to the locals queue instead. The "type" column can be set to "regex", "fnmatch", "substr" and "exact".
If you want to send mail from a different IP address than is stored in the message JSON, you can use the "delegates" table. For example, if you notice that "example.com" no longer accepts incoming connections from your "1.2.3.4" address, but that it does accept connections from "1.2.3.5", you can add a row to the "delegates" table. This row instructs MailerQ that it should send all mail that was originally planned to be sent from "1.2.3.4" to "example.com" to be sent from address "1.2.3.5" instead.
To pause deliveries from and/or to IP addresses or domains, you can insert records in the "paused" table. The "target" column in this table can hold IP addresses, domain names and empty values. The "localip" column is of course only used for IP addresses, but may also hold empty strings.
If you want to pause all deliveries from your own IP address "1.2.3.4" to "example.com", you can simply insert such a record in the "paused" table. An empty value in one of the columns of the "paused" table means that all deliveries are paused. For example, if you insert an empty domain and IP address "1.2.3.4", MailerQ pauses all deliveries from "1.2.3.4", no matter to what domain they should be sent. If you insert a record with an empty domain and IP "5.6.7.8" as target, all deliveries to IP address "5.6.7.8" will be put on hold.
By default, MailerQ does a reverse DNS lookup to find the hostname of its local IP addresses. This hostname is the name used in the "HELO/EHLO" SMTP handshake. If you want to use a different hostname instead, you can insert a row into this table.
To override the DNS resolver, you can add records to the "dns" table. MailerQ normally uses normal DNS queries to find out to which IP addresses mail should be sent. First it does an MX query to find the MX records, and then for each MX record MailerQ looks for the associated A or AAAA records to find the IP addresses.
However, in case you want to bypass these lookups for specific IP addresses, you can add records to the "dns" table. If matching records are available, MailerQ uses the IP addresses in these records instead of doing the DNS queries. This feature can for example be useful if you know that certain domains (like the ones that you own yourself) have special IP addresses on which they receive email, besides the ones that are announced in DNS.
The easiest way to try out MailerQ is using Docker. The MailerQ docker image is completely self-contained, and allows you to run MailerQ without the hassle of setting up its dependencies in under a minute.
First, to install the latest version of the MailerQ image, run
docker pull mailerq/mailerqNow, create a trial if you have not already done so. Then, the simplest method is to copy your license key from here and supply it to Docker inside an environmental variable. To do this, you can run
docker run -e LICENSE_KEY=<your_license_key> -it mailerq/mailerqwith your license key filled in. This way, the image will automatically download a short-lived license on each startup, ideal for testing.
Alternatively, if you already have a license.txt file, you can bind the image license
file to the host file. To accomplish this, you can run
docker run -v `pwd`/license.txt:/etc/mailerq/license.txt -it mailerq/mailerqwith the license.txt in the current working directory.
That is it! Now you can access the MailerQ management console here and the RabbitMQ management console here.
By default, the image runs on its own network managed by Docker, which means that the
default username and password guest in RabbitMQ do not work. Therefore, on startup,
an extra account mailerq is created with password set to mailerq.
The easiest way to get your hands on the software is to add the Copernica repository to the source list of your package manager. You can then use "apt" package manager to install MailerQ and to get updates. If you do not want to use your package manager, you can download MailerQ manually. Check out our downloads page for a list of available files that can be manually installed.
MailerQ comes in two separate packages which are available on one of our repositories. Each supported operating system has its own repository.
| Debian 12 (bookworm) | deb https://packages.mailerq.com/debian bookworm main |
| Debian 11 (bullseye) | deb https://packages.mailerq.com/debian bullseye main |
| Debian 10 (buster) | deb https://packages.mailerq.com/debian buster main |
| Ubuntu 22.04 (jammy) | deb https://packages.mailerq.com/debian jammy main |
| Ubuntu 20.04 (focal) | deb https://packages.mailerq.com/debian focal main |
| Ubuntu 18.04 (bionic) | deb https://packages.mailerq.com/debian bionic main |
First add the MailerQ repository to your system. This is the repository from which all MailerQ software can be downloaded. For Ubuntu 22.04, Debian 11 and newer Debian based distros you can use the following instructions:
# download and install the gpg key
sudo gpg --homedir /tmp --no-default-keyring --keyring /usr/share/keyrings/mailerq-packages.gpg --keyserver keyserver.ubuntu.com --recv-keys 90DE523265E3563E
# find out the distro name
MY_DISTRO=`lsb_release -c -s`
# add the repo
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/mailerq-packages.gpg] https://packages.mailerq.com/debian $MY_DISTRO main" | sudo tee /etc/apt/sources.list.d/mailerq-packages.listOlder versions of Debian and Ubuntu still support apt-key, which is less secure, but also simpler:
# download and install the gpg key
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 90DE523265E3563E
# add the repo
sudo apt-add-repository https://packages.mailerq.com/debianThe MailerQ frontend and backend are distributed in separate packages, so that bugfixes to the frontend can be brought out without forcing a restart of MailerQ. Both packages can be installed independently:
# make sure apt definitions are up-to-date
sudo apt update
# install the latest MailerQ-server
sudo apt install mailerq-server
# install the latest mailerq-frontend
sudo apt install mailerq-frontendNew versions are automatically installed every time that you run "apt ugrade":
# make sure the local apt cache is up-to-date
sudo apt update
# get upgrades for all installed packages
sudo apt upgradeAll repositories mentioned above contain MailerQ versions that are statically linked, which means that there are not many dependencies. However, due to licensing conditions, we are required to supply dynamically linked versions of MailerQ too. If you prefer using a dynamically linked version, you can send an email to info@mailerq.com to request the specific dynamically linked version you need.
If you want to retrieve the recipient from a MQ_Message, you can use this function for it. It returns a NULL terminated string, or simply NULL if no recipient has been set.
/**
* Retrieve the recipient
*
* @param message the message to retrieve the recipient from
* @return the recipient
*/
const char *MQ_recipient(MQ_Message *message);This function has a different behavior if you call it on a message that is being received and on a message that is being sent. If you call it on a message that is being received, for example inside your mq_smtp_in_message() function, it is simply an alias for retrieving the "recipient" property from the JSON.
For outgoing messages however, the recipient set in the JSON and the recipient that is returned by this function may be different. This function returns the recipient that is actually being used in the SMTP protocol for the "RCPT TO" instruction, and could be different than the one that was originally loaded from RabbitMQ (for example, when a plugin modified the recipient in the mean time).
For more info, see the documentation about MQ_setRecipient.
MailerQ has a non-blocking nature, and uses callbacks to pass control to a plugin. It is then up to the plugin to interact with the event loop, and pass control back to MailerQ when it is ready with its task. Passing back control can be done in three different ways:
The MQ_complete() implements the second behavior. If you call it, MailerQ will skip all subsequent plugins and continues with its original algorithm.
/**
* Finish plugin processing
*
* @param connection the connection that may move on to the next plugin
*/
void MQ_continue(MQ_Connection *connection);See also MQ_continue() and MQ_retry() for the other ways of handing back control.
All delivery attempts like received emails, downloads and errors can be logged by MailerQ in different log files. The log file settings can be specified in the "/etc/mailerq/config.txt" config file.
In previous version of MailerQ (< 5.6), log rotation was handled in the application itself. This is not very flexible, and has therefore been removed. Instead, log rotation is now delegated to logrotate. The packaged config file for logrotate looks as follows. For specific settings, or different filenames, see the man page.
#
# Logrotate config file for MailerQ
#
# This file contains settings to automatically rotate the logfiles
# using logrotate. To see all possible settings, use `man 8 logrotate`.
#
# Settings for all the user formatted logfiles. If you renamed logfiles, or the files
# are in a non-standard location, alter the glob below.
/var/log/mailerq/*.log {
# General settings, may be altered.
rotate 20
daily
compress
missingok
notifempty
maxsize 100M
# Required setting for normal logfiles, because MailerQ monitors the filesystem
# and needs time to close the writers to a logfile
delaycompress
}
# Settings for all the binary logfiles. If you renamed logfiles, or the files
# are in a non-standard location, alter the glob below.
/var/log/mailerq/*.bin {
# General settings, may be altered.
rotate 20
daily
missingok
notifempty
maxsize 100M
# Required setting for binary logfiles, because MailerQ reads out the logfiles for the
# web interface and processing purposes.
nocompress
}Note that logrotate is not required. In fact, any application can simply remove the logfiles, and MailerQ opens it almost immediately. Naturally, if an application removes a logfile, the application should wait until MailerQ has finished writing and opened a new logfile.
Information about MailerQ is logged to the application log. This includes failures like databases that are suddenly offline or RabbitMQ connections that are lost. Currently the log levels Info, Warning and Critical are supported. The error log has been deprecated in favor of the application log. The application log can be set using the option application-log, and supports logging to stdout, stderr, syslog and files. If this setting is kept empty no logs are stored.
application-log: syslog,/var/log/mailerq/mailerq.logFiles must, of course, be writable for MailerQ.
You can use the Smart-TPL templating language to define your own log lines. With this template language, you can use some simple logic to log specific errors. You can specify this in the configuration file.
Example configuration:
send-log-format: [{$date}] {$result.state}:{$result.result} {$attempt} <{$email.envelope}> <{$email.recipient}> [{$result.from}] [{$result.to}] {$email.tags} {$result.code} {$result.status} {$result.description}Example output:
[2018-08-29 07:20:16] message:accepted 1 <info@company.com> <info@outlook.com> [100.20.20.20] [100.30.30.30] [ "a", "b" ] 250 2.0.0 Ok (edited)You can only use the Smart-TPL templating language for the send-log-format, received-log-format, download-log-format and the www-log-format settings.
You can only create your own log lines with the available variables and properties. Access to variables and properties is available via this syntax:
Access to variable variable = {$variable}
Access to property variable.property = {$variable.property}
With the help of logical constructors, you can log specific errors.
{if $variable}Print:{$variable} {else}show something else{/if}For more information about this templating language, you can read the Smart-TPL documentation.
MailerQ creates log files for all delivery attempts. The directory in which these log files are stored can be set here, as well as the name of the log file.
The management console web interface uses binary log files to display real-time information of all delivery attempts.
The configuration file holds the following options:
# Log of send attempts
send-bin-log: /var/log/mailerq/send.bin
send-log: /var/log/mailerq/send.log
send-log-format: [{$date}] {$result.state}:{$result.result} {$attempt} <{if $email.envelope}{$email.envelope}{/if}> <{$email.recipient}> [{$result.from}] [{$result.to}] {$email.tags} {$result.code} {$result.status} {$result.description}You can specify your own log lines via the send-log-format setting. These are the available variables.
$email Email JSON object, as placed in queue.
$result Last result object, of this connection
$attempt int, attempt number of this mail
$date formatted DateTime (e.g. 01-01-1970 11:53)The available properties are specified in the JSON for outgoing messages.
The "send-log" is the location for the send log. The default is "send.log". Its directory must be writable for MailerQ.
Incoming messages - messages sent to the SMTP port of MailerQ - are logged to the received log file. These settings work exactly like the send log settings:
# Log of incoming messages
received-bin-log: /var/log/mailerq/received.log
received-log: /var/log/mailerq/received.log
received-log-format: [{$date}]{if $json.message_id} {$json.message_id} {/if}<{$json.envelope}> <{$json.recipient}> [{$json.connection.remote_ip}] [{$json.connection.local_ip}]You can specify your own log lines via the received-log-format setting. These are the available variables.
$json The received / constructed JSON
$report (boolean) whether or not this is a delivery report
$date The datetime
$time unix timestampThe available properties are specified in the JSON for incoming messages
The log file only holds messages that are received over the SMTP port. Messages dropped in the spool directory or that are injected using the command line interface are not logged.
MailerQ can create MIME message itself. To do this, MailerQ sometimes has to download resources from the internet (like images that should be embedded, or attachments that should be included in the mail). All these download operations are written to the download log.
The settings for the download log are essentially the same as the settings for the send-log:
# Log of downloads
download-bin-log: /var/log/mailerq/download.bin
download-log: /var/log/mailerq/download.log
download-log-format: {$date}, {$time}, {$url}You can specify your own log lines via the download-log-format setting. These are the available variables.
$url The url
$time The unix timestamp
$date The formatted DateTime (e.g. 01-01-1970 11:53 oid)You can monitor who accessed the management console. The settings for the management console log are essentially the same as the settings for the send-log:
# Log of management console
www-bin-log: /var/log/mailerq/download.bin
www-log: /var/log/mailerq/
www-log-format: {$date}, {$ip}, {$request}You can specify your own log lines via the www-log-format setting. These are the available variables.
$date The formatted DateTime (e.g. 01-01-1970 11:53 oid)
$ip The IP that connected
$request A string with the request (e.g. "GET/settings")From MailerQ 5.6 and onward, user formatted logs can be combined. Simply specify the same filename as a target for multiple logs. For example, to combine the user formatted send/receive logs, the config settings below can be used.
send-log: /var/log/mailerq/combined.log
received-log: /var/log/mailerq/combined.logThis only works for a single instance. If multiple instances point to the same file, no guarantees about data integrity will be made. Same holds for previous versions on MailerQ (< 5.6), there was no logic to guarantee data integrity even on the same instance. This works fine for smaller loglines, but it depends on the platform and the size of the logged information when this corruption will occur.
MailerQ writes all events to RabbitMQ message queues. The recommended way of handling events is therefore to write scripts or applications that process the data from these message queues. This is much more powerful than periodically processing log files:
For all of the above reasons, the first MailerQ versions did not even have logging capabilities. We did not want to slow down our high-performance MTA by having it write data to disk - it was up to other scripts and applications to read the results from the message queues, react to them and write data to appropriate log files. With this architecture, hiccups in disk IO could not slow down email deliveries.
However, although this worked (and works!) perfectly for us, we found out that many users still want to have log files to monitor what is going on. So we've added the logging feature, and we use threads to prevent that IO hiccups can be a problem. But keep in mind that once you find yourself writing cron jobs to process log files, you would probably be better off writing a script that processes data from the message queues.
You can instruct MailerQ to use a "smart host". When you enable this feature, MailerQ does not send the messages directly to the final recipient, but to an alternative mailserver instead: the smarthost.
This feature could for example be useful if you use MailerQ for sending mails from your own server to the internet, but you want to use an external service (like smtpeter.com) to track the clicks and opens for your mailings. Other use cases for smarthosts are when you want to create a chain of MailerQ instances, and have the mails from one instance forwarded to the next instance.
By default, the smarthost feature is disabled. If you want to enable it, you either need to explicitly configure it in the configuration file, or you can enable it on a per-message level. The following config file variables can be used to enable the smarthost for all messages that flow through MailerQ:
smarthost: <hostname> (empty by default)
smarthost-port: <port> (default: 25)
smarthost-username: <username> (empty by default)
smarthost-password: <password> (empty by default)The "smarthost" has to be set to enable the smarthost feature, and should contain the hostname of the server that you want to redirect the email to. The other "smarthost-*" options are optional, and can be used if the target smarthost uses a different port than the default one (default is 25), and if the smarthost requires authentication.
If you do not want to send all messages using a smarthost, but only specific messages or send specific messages to specific smarthosts, you can do so by adding the smarthost variables to the JSON or MIME-header of a specific message. If global smarthost settings are set, these settings will override the settings set in your configuration file.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "...",
"smarthost": {
"hostname": "hostname-smarthost",
"port": "25",
"username": "example-user",
"password": "example-password"
}
}You can use MIME headers too to enable the smarthost on a per-message level. The following header variables have the same meaning as the JSON settings mentioned above:
x-mq-smarthost-hostname: <hostname-smarthost>
x-mq-smarthost-port: <25>
x-mq-smarthost-username: <example-user>
x-mq-smarthost-password: <example-password>Only the "hostname" property is mandatory. The port defaults to 25, and not setting the username or password means MailerQ will not try to authenticate to the smarthost.
If you use the "smarthost" option mails are not delivered to the actual recipient but to the smarthost server. As a consequence, MailerQ internally queues all mails as if they were sent to the smarthost domain. If you open the management console, the list of active domains will contain only one item: the smarthost domain indicated by "(smarthost)", which is where all emails are going to be sent to.
If you are debugging, you often want to run MailerQ in (almost) the same configuration as you normally would, with queues for all the domains to which messages are being delivered. The only difference is that you do not want to actually deliver the mails, but send them to a dummy destination: the "smarthost". This is where the smtp-sink options comes in.
smtp-sink-address: <ip address or hostname> (default: 0.0.0.0)
smtp-sink-port: <port> (default: 25)
smtp-sink-username: <username> (empty by default)
smtp-sink-password: <password> (empty by default)If you include the "smtp-sink-address" and "smtp-sink-port" options in the config file, MailerQ runs normally, and all mails are routed through different internal queues with their own send capacities. However, when a TCP connection is created, the connection will be set up to the sink instead of to the actual recipient mail server. Bear in mind that if a hostname is provided, it will only be resolved once, at the start of the application, and the resolved IP will be used until the application is stopped.
The "smtp-sink-username" and "smtp-sink-password" options can be used if your sink requires authentication.
If you are looking for an SMTP server that simply accepts and discards incoming messages, you can use Postfix' smtp-sink.
It is possible to configure both the smtp-sink option and the smarthost at the very same time. If you do this, MailerQ will keep a single queue of messages to the smarthost (and thus not separate queues for hotmail, gmail, yahoo, et cetera) - but when the TCP connection is set up, it does not connect to this smarthost, but to the sink address instead.
When MailerQ receives an incoming SMTP connection, it will accept that connection, and iterate over all loaded plugins, and call the mq_smtp_in_connect() function in each of these plugins.
You can implement this function in your plugin if you want to run certain code right after an incoming TCP connection comes in on the SMTP port of MailerQ. Your function should return true if you want your plugin to take over control from MailerQ, or false if control should stay with MailerQ.
bool mq_smtp_in_connect(MQ_Context *context, MQ_Connection *connection);
The function takes two parameters:
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
This structure is provided for every asynchronous function a plugin can implement. It can be used to get access to the event loop as well as indicating to MailerQ when the plugin is done.
MailerQ is a multi-threading application, with multiple working threads that are all sending and receiving messages at the same time. For every single worker thread, a separate context object is created. If you want to be notified when a new context is created, or when a context is destroyed, you can define the following functions in your plugin:
| Implementable function | Description |
|---|---|
| mq_context_initialize() | Called by MailerQ when a worker thread is created |
| mq_context_cleanup() | Called by MailerQ when a worker thread is destroyed |
Both of the listed functions are optional. It is not necessary for your plugin to export them, but you can for example use these functions to initialize plugin specific data, and to deallocate that data when the worker thread exits.
Your plugin may store a pointer to plugin specific data in a context. This is a pointer that is completely ignored by MailerQ, but that might be useful for your plugin. If you allocate data for this, it is also your responsibility to deallocate the data when the context is destroyed.
| Callable function | Description |
|---|---|
| MQ_contextData() | Retrieve the plugin managed data from the context |
| MQ_setContextData() | Assign plugin managed data to the context |
The plugin data is not shared between plugins. Every plugin can therefore associate its own data with a context, and you do not have to worry about other plugins that overwrite your data.
The main feature that a MQ_Context offers, is access to the event loop. There are a number of functions available that you can use for registering file descriptors and timeouts with the event loop. Internally, MailerQ uses a libev event loop, and you can also get access to this loop pointer.
| Callable function | Description |
|---|---|
| MQ_ev() | Access to the underlying libev instance |
| MQ_ioWatch() | Watch a filedescriptor for activity |
| MQ_ioUnwatch() | Stop watching a filedescriptor for activity |
| MQ_startTimer() | Start a timer |
| MQ_stopTimer() | Stop a timer |
| MQ_resetTimer() | Reset a timer to a new timeout |
This function will start monitoring a file descriptor for the requested events. When one (or more) of these events occur, the given callback will be invoked.
/**
* Monitor a file descriptor for activity
*
* @param context the context to register the file descriptor in
* @param fd the file descriptor to monitor
* @param events the events to monitor, binary or'ed MQ_READ and/or MQ_WRITE
* @param callback callback to invoke when one or more of the events occur
* @param data custom data to supply to the callback
*/
MQ_IOWatcher *MQ_ioWatch(MQ_Context *context, int fd, int events, void(*callback)(MQ_IOWatcher *watcher, int fd, int events, void *data), void *data);As an example, we will read from the standard input until an empty line is found, at which point we will stop listening to the input. This example makes no sense for a MailerQ plugin, but will illustrate the point.
#include <mailerq.h>
#include <string.h>
#include <stdio.h>
/**
* Callback function to receive standard input
*
* @param watcher the watcher that is monitoring the file descriptor
* @param fd the file descriptor that received events
* @param data custom data pointer
*/
void callback(MQ_IOWatcher *watcher, int fd, int events, void *data)
{
// buffer for data to process
char data[1024];
ssize_t length;
// we are working with STDIN, which is never writable
// (because that's what we have STDOUT for), so we don't
// have to check the events variable, otherwise we could
// check for readability or writability like this:
//
// events & MQ_READ
// events & MQ_WRITE
// normally, when we check for readability we should not
// assume that we are receiving complete lines, however
// STDIN is line-buffered, so the callback is only invoked
// whenever the user presses return after input
// read data from input
length = read(fd, data, sizeof(data));
// if there are only two characters we received an empty line
// since the line includes the CRLF produced by the terminal
if (length == 2)
{
// stop watching the file descriptor
MQ_ioUnwatch(watcher);
}
else
{
// process the data here
}
}
/**
* Monitor standard input for data
*
* @param context the context to monitor in
*/
void monitor(MQ_Context *context)
{
// monitor stdin for input
MQ_ioWatch(context, STDIN, MQ_READ, callback, NULL);
}After a user has sent its login credentials over a SMTP connection, MailerQ will notify all the plugins about this, and call the mq_smtp_in_authenticate() function in each of the plugins (only if the function exists, of course).
Most plugins implement this function to add their own login administration system to MailerQ. If authentication succeeds, you can call MQ_setAuthenticated() to mark the connection as being an authenticated connection.
This function should return false if MailerQ can stay in control, or true if the plugin takes over control (for example because it starts a non-blocking database lookup). If the plugin takes over control, it should of course give back control to MailerQ when the credentials lookup has been completed.
bool mq_smtp_in_authenticate(MQ_Context *context, MQ_Connection *connection, const char *login, size_t loginsize, const char *passwd, size_t passwdsize);
If your plugin returns true, MailerQ hands over control to your plugin, and you should also hand control back. For more information on how MailerQ plugins can interact with the event loop, and how control is passed to and from plugins, see the article about the MailerQ event loop.
If you want to retrieve the envelope from a MQ_Message, you can use this function for it. It returns a pointer to a MQ_Address structure, or simply NULL if no envelope was set.
/**
* Retrieve the envelope address
* @param message the message to retrieve the envelope address from
* @return the envelope
*/
MQ_Envelope *MQ_envelope(MQ_Message *message);This function has a different behavior if you call it on a message that is being received and on a message that is being sent. If you call it on a message that is being received, for example inside your mq_smtp_in_message() function, it is simply an alias for retrieving the "envelope" property from the JSON.
For outgoing messages however, the envelope set in the JSON and the envelope that is returned by this function may be different. This function returns the envelope that is actually being used in the SMTP protocol for the "MAIL FROM" instruction, and could be different than the one that was originally loaded from RabbitMQ (for example, when a plugin modified the envelope in the mean time).
For more info, see the documentation about MQ_setEnvelope.
In the end, MailerQ fetches all messages that it is going to send directly from a RabbitMQ outbox queue. The fastest and most efficient way to inject emails is therefore to bypass SMTP, spool directories or the command line interface, and publish messages directly to RabbitMQ.
If you want to publish messages directly to RabbitMQ, you need a client library to communicate with RabbitMQ. On the RabbitMQ website you can find many plugins and libraries for this.
Besides being faster, publishing messages directly into the outbox queue has the advantage that you can set all sorts of special properties in the JSON objects that control the delivery.
The JSON encoded messages that you add to the outbox message queue should contain a minimum of two properties that hold the recipient's email address, and the full MIME message. All other properties are optional.
{
"recipient": "info@example.org",
"mime": "From: my-sender-address@my-domain.com\r\n
To: info@example.org\r\n
Subject:Example subject\r\n\r\n
This is the example message text"
}If you're interested in bounces and out-of-office replies, we recommend to also include an "envelope" property, because that's the address where bounces will be sent to.
{
"envelope": "my-sender-address@my-domain.com",
"recipient": "info@example.org",
"mime": "From: my-sender-address@my-domain.com\r\n
To: info@example.org\r\n
Subject:Example subject\r\n\r\n
This is the example message text"
}Note that for ease of reading we added some spaces to the message mimes in the above examples. The above examples are very minimalistic, check the JSON message specification for a full list of all supported properties.
It is possible to compress the data that you send to MailerQ. The JSON message that you send to RabbitMQ can be gzip compressed. If you do this, you also have to add a "content-encoding" header to the AMQP envelope in which the message is wrapped, and set it to the value "gzip". Check the documentation of your AMQP library to find out how to set properties in the AMQP envelope.
The MailerQ program can be started in two different modes: as a daemon process that runs all the time and that sends out the messages, or as a command line utility that reads a message from standard input and publishes it to the inbox queue. As a command line utility, MailerQ does not attempt to send out the mail, it just puts the message in the RabbitMQ message queue. You need a separate running MailerQ daemon process that takes care of the delivery.
When you normally start up MailerQ, it operates as a daemon process. To start
it as a command line utility for reading a message from standard input, you need
to pass extra arguments to the program; either the email addresses of the recipients,
or the option --extract-recipients to tell MailerQ that it should read in a
mime message from standard input and filter out the destination addresses from it.
$ mailerq recipient1@example.com recipient2@example.com
$ mailerq --extract-recipients
The message read in by MailerQ is converted into a JSON encoded email message and published to the inbox queue. Another MailerQ daemon process can then pick it up from there and take care of the delivery.
MailerQ reads in all data from standard input until it encounters a line with a single dot on it. It treats this line as the end-of-message marker and publishes all data up to this dot (but not including the dot) to the inbox message queue. This is the same behavior as "sendmail" and allows you to use MailerQ as drop-in replacement for sendmail.
This behavior could however be problematic if your email contains lines that start with a dot. Such lines should of course not be treated as end-of-message markers. There are two ways to prevent this; you can either stuff those lines with an extra dot in front. MailerQ will automatically recognize this and removes the extra dot, or you can add the command line option "--ignore-dot" to instruct MailerQ that dots do not have a special meaning, and that the message is ended with end-of-file instead.
$ mailerq --ignore-dot recipient1@example.com recipient2@example.com$ mailerq --ignore-dot --extract-recipientsThe envelope address that MailerQ uses for sending out an email is automatically extracted from the message MIME. It can be set in the custom "x-mq-envelope" header, or in the "return-path", "sender" or "from" header (in that specific order). However, you can also set the envelope address manually with the "--envelope" command line option:
$ mailerq --envelope myaddress@mydomain.com --extract-recipients
You can use regular unix shell pipes and I/O redirection operators to get the contents from a file into MailerQ.
$ cat email.txt | mailerq --extract-recipients --ignore-dot$ mailerq --extract-recipients --ignore-dot < email.txt This article contains a list of all configuration settings, sorted alphabetically. Each setting links to a documentation article that explains the setting more in-depth. Or do you want to search by topic?
| Setting | Description |
|---|---|
| application-log | Full path to the application log |
| cluster-address | Address for a cluster to share between instances |
| cluster-exchange | Set cluster exchange |
| cluster-timeout | Number of seconds the --list-cluster option waits |
| cluster-verify | Should the server certificate be verified (only used for amqps:// connections) |
| database-address | Relational database for config data and delivery settings |
| database-ttl | Time to live: Database reload interval (Default 600s) |
| database-validate | Check and fix database settings on startup |
| download-blacklist | IP addresses from which no resources can be downloaded |
| download-cache | Address of the cache (Used in ResponsiveEmail) |
| download-log-format | Download log format |
| download-log | Download log filename |
| download-ttl | Max time to keep downloaded resources in the cache (Used in ResponsiveEmail) |
| download-whitelist | Exceptions to the download-blacklist |
| download-proxy | Route outgoing download requests through a proxy |
| dsn-advertise | Toggle announcement about DSN extension and parameters |
| dsn-notify | Events that trigger a notify |
| dsn-ret | Send full mail (FULL) or headers (HDRS) for notifications |
| dns-min-ttl | Minimum TTL to respect when doing domain lookups |
| heartbeat-enabled | Toggle user statistics (default: true) |
| license | MailerQ license file |
| lock | Process ID for MailerQ instance to prevent from starting instances more than once |
| max-attempts | Maximum attempts to deliver email |
| max-delivertime | Maximum time spent trying to deliver email |
| plugin-directory | Directory with MailerQ plugins |
| proxy-server | Setup proxy for outgoing HTTP requests |
| rabbitmq-address | Location and authentication to connect to RabbitMQ |
| rabbitmq-consumers | Amount of consumer threads (default: 1) |
| rabbitmq-declare | Enable or disable checking and declaring the queues on startup |
| rabbitmq-dsn | Your RabbitMQ delivery status notification queue |
| rabbitmq-durable | Durable/not durable RabbitMQ queues (default: true) |
| rabbitmq-encoding | Choose encoding for messages (such as gzip) |
| rabbitmq-exchange | RabbitMQ exchange |
| rabbitmq-failure | Your RabbitMQ failure queue |
| rabbitmq-inbox | Your RabbitMQ inbox queue |
| rabbitmq-local | Your RabbitMQ local queue |
| rabbitmq-lazy | Lazy/non lazy RabbitMQ queues (default: non-lazy) |
| rabbitmq-maxpriority | Enable the use of priority queues |
| rabbitmq-qos | Set a maximum number of messages simultaneously consumed per consumer thread |
| rabbitmq-minqos | Set a minimum number of messages simultaneously consumed per consumer thread |
| rabbitmq-flush-qos | Set a maximum number of messages simultaneously consumed during flush |
| rabbitmq-outbox | Your RabbitMQ outbox queue (must be unique if multiple instances of MailerQ are used) |
| rabbitmq-persistent | Persistent/not persistent RabbitMQ queues (default: false) |
| rabbitmq-publishers | Amount of publisher threads (default: 1) |
| rabbitmq-queues | Names of the RabbitMQ queues to manage email (inbox, failures, retries, etc...) |
| rabbitmq-refused | Your RabbitMQ refused queue |
| rabbitmq-reports | Your RabbitMQ report queue |
| rabbitmq-results | Your RabbitMQ result queue |
| rabbitmq-retry | Your RabbitMQ retry queue |
| rabbitmq-success | Your RabbitMQ success queue |
| rabbitmq-verify | Should the server certificate be verified (only used for amqps:// connections) |
| received-log-format | Received log format |
| received-log | Received log filename |
| retry-interval | Interval for retrying to send |
| send-bin-log | Binary send log filename |
| send-log | Send log filename |
| send-log-format | Send log format |
| server-id | Server ID to prevent multiple instances from assigning same message ID |
| smarthost | Smarthost name (only needed if Smarthost is desired) |
| smarthost-password | Smarthost password |
| smarthost-port | Smarthost port |
| smarthost-username | Smarthost username |
| smtp-auth | Authentication string for authenticating to SMTP server |
| smtp-auth-results | Toggle adding authentication results to MIME header |
| smtp-certificate | Certificate for secure SMTP connections |
| smtp-check | Comma separated list of checks to perform on incoming messages |
| smtp-ciphers | Supported ciphers for secure SMTP connections |
| smtp-connections | Maximum amount of open TCP connections allowed |
| smtp-default-ips | List of default outgoing IP addresses |
| smtp-extract | Toggle automatic extraction of metadata |
| smtp-hostname | The hostname that clients see when connecting to the SMTP server |
| smtp-ip | SMTP server IP |
| smtp-key | Private key for secure SMTP connections |
| smtp-mappable-ips | List of mappable IP adresses |
| smtp-maxsize | Maximum message size to accept |
| smtp-port | Ports for incoming MailerQ connections |
| smtp-proxy | IPs to treat as connection from proxy server |
| smtp-range | Range of IP address to allow incoming email for |
| smtp-secure port | Ports for secure incoming MailerQ connections |
| smtp-sink-address | SMTP sink domain name or ip address |
| smtp-sink-password | SMTP sink password |
| smtp-sink-port | SMTP sink port |
| smtp-sink-username | SMTP sink username |
| smtp-threads | Number of threads started for SMTP traffic |
| smtp-timeout | Max idle time on incoming SMTP connections |
| smtp-protection | Enable SMTP incoming IP blacklist |
| smtp-unmappable-ips | List of unmappable IP addresses |
| spamassassin-host | Address for SpamAssassin daemon |
| spamassassin-port | Port for SpamAssassin daemon |
| spool-delay | Delay for processing spool directory files |
| spool-directory | Path to the directory to monitor |
| spool-extract | Toggle scanning for headers in spool directory mail |
| spool-fail-directory | Path to move files to that could not be picked up |
| spool-open-files | Amount of files that may be open simultaneously by the spool directory |
| spool-remove | Toggle removal of files from spool directory |
| spool-threads | Amount of threads to use for the spool directory |
| storage-address | Address for external message storage |
| storage-policy | Message store policy |
| storage-reschedule | Number of seconds to wait before a mail is retried |
| storage-threads | Storage threads |
| storage-timeout | Timeout for fetch operations |
| storage-ttl | Time to live: Storage reload interval |
| user | Change user for MTA |
| validate-envelope | Validate envelope address before sending |
| validate-recipient | Validate recipient address before sending |
| www-auth | Management console authentication string |
| www-cache-control | Enable or disable browser cache |
| www-certificate | Certificate file for secure connection to management console |
| www-ciphers | Supported ciphers for secure management console |
| www-connections | Limit for simultaneous HTTP connections to built-in HTTP server |
| www-ip | Web interface IP (IP/port combination unique for every instance of MailerQ) |
| www-dir | Directory for installing MailerQ management console files |
| www-host | Web interface host (IP/port combination unique for every instance of MailerQ) |
| www-port | Port number for the management console |
| www-private-key | Private key for secure connection to management console |
| www-secure-port | Secure (HTTPS) port number for the management console |
| www-url | The URL via which the management console is accessible |
To check your version of MailerQ for example, just type in the cli:
$ mailerq --version| Setting | Description |
|---|---|
| daemon | Deamon process (deafault true) |
| disable-crash-report | Disable crash report |
| disable-crash-post | Disable automatic crash posting |
| extract-recipients | Read in a mime message from standard input and filter out the destination addresses |
| ignore-dot | Instruct MailerQ that dots do not have a special meaning |
| list-ips | List IP addresses |
| loopback | Loopback |
| notify-cluster | Notify cluster |
| paused | Pause |
| purge-database | Rebuilding the database tables |
| repair-database | Repair the database |
| version | Version |
MailerQ relies on external RabbitMQ message queues for message queueing. All incoming and outgoing emails are read from and published to RabbitMQ queues, and all results are published to such queues as well.
The messages in the queues all are JSON formatted. This means that if you develop scripts or programs that inject emails directly into a message queue, your messages must be JSON encoded as well, and must use properties that are understood by MailerQ. Scripts or programs that process results from the result queues, should be able to read in and understand the JSON formatted result objects.
If you inject mails in MIME format (for example via the SMTP interface, the spool folder or the command line interface) you can of course not make use of the JSON properties to control the delivery. The properties that you would otherwise set in the JSON can then be set using special MIME headers that can be added to the email message.
All config file settings that start with rabbitmq-* control how MailerQ interacts with RabbitMQ. Amongst these settings are the address of the RabbitMQ server, the queue from which outgoing messages are read, and the queues to which incoming messages are delivered.
MailerQ reads the location and authentication information to connect to RabbitMQ from its config file. This data is stored in the "rabbitmq-address" setting:
rabbitmq-address: amqp://login:passwd@hostname/vhostThe format of the address is obvious: the "login" and "passwd" hold the username and password of the RabbitMQ server, the hostname is the name of the server on which RabbitMQ runs, and the optional vhost is the name of the "virtual host" inside RabbitMQ that you have reserved for all MailerQ related data.
If you leave the "rabbitmq-address" setting empty, MailerQ uses the "amqp://guest:guest@localhost/" default value. This default only works if you run RabbitMQ and MailerQ on the same server, and when you do not use a special vhost.
If you have a cluster of RabbitMQ nodes, you can use commas to separate the addresses:
rabbitmq-address: amqp://guest:guest@host1/vhost,amqp://guest:guest@host2/vhost,amqp://guest:guest@host3/vhostRunning a cluster allows you to use highly available queues.
If your RabbitMQ server supports secure connections, you can configure MailerQ to connect to an "amqps://" address instead. The communication between MailerQ and RabbitMQ will then be encrypted.
rabbitmq-address: amqps://guest:guest@hostname/vhost
rabbitmq-verify: falseMailerQ checks the server certificate when it connects to a secured RabbitMQ server. If this certificate can not be verified with one of the known openssl certificate authorities, MailerQ refuses to set up a connection. If you use a self-signed certificate, you may want to skip this extra test, and set the config file option "rabbitmq-verify" to "false".
RabbitMQ supports priority queues. If you use a priority queue for the outbox, messages with a higher priority are consumed by MailerQ before messages with a lower priority, and are therefore also sent out faster. This could be useful if you want to inject messages that should be sent before emails already queued.
To turn a queue into a priority queue, the queue in RabbitMQ must have been declared with the "x-max-priority" property set. If you declare your own queues, you have to do this yourself. If you rely on MailerQ to declare the queues, you can specify the value of this property in the config file:
rabbitmq-maxpriority: 4The above config file option instructs MailerQ to set the "x-max-priority" property on the queues it creates. This setting is used during application startup when the queues are declared, and when new temporary queues are declared. If queues already exist, they must match the "x-max-priority" setting of the existing queues.
The reason MailerQ declares all queues to be priority queues is so that messages that are higher priority can be picked up sooner in every part of the chain, also its reports and its results. The same holds for messages that are injected, as they can also have a priority setting set, allowing for injected messages also to be picked up sooner.
In general, this number should be kept as small as possible, as there is a significant overhead for priority queues. For example, when a priority queue is created with an "x-max-priority" of 4, RabbitMQ internally creates 4 queues. As this happens for every queue MailerQ declares, priority queues with more than two or three priority levels are generally discouraged.
All queues and the exchange are automatically created when MailerQ starts. If you do not want MailerQ to declare the queues and exchanges, you can use the "rabbitmq-declare" setting in the config file. If you set this to false, no queues or exchanges are declared on startup.
rabbitmq-declare: falseDisabling the declaration of queues and exchanges is helpful if you have already created them yourself and you do not want MailerQ to do this for you. If you have created the queues with slightly different settings than MailerQ does, you even need this setting because otherwise MailerQ reports a queue incompatibility error and refuses to start.
In the MailerQ configuration you can specify whether you want queues to be durable and whether you want messages to be persistent using the following two options:
rabbitmq-durable: true (default: true)
rabbitmq-persistent: false (default: false)
rabbitmq-lazy: false (default: false)MailerQ creates several queues and exchanges in RabbitMQ. When MailerQ starts, it first checks if the queues and exchanges that you have configured in the MailerQ config file exist. If they do not, MailerQ sends instructions to RabbitMQ to create the required exchanges and queues. RabbitMQ allows you to mark your queues and exchanges to be "durable". This means that the exchange or queue will continue to exist even when RabbitMQ is restarted. In theory it is a slightly better to enable durable queues (but you probably won't notice much of a difference because the queues and exchanges are automatically re-created on startup anyway).
The "rabbitmq-persistent" setting toggles whether messages published to RabbitMQ should be cached in main memory or on disk. With the default "false" setting, RabbitMQ keeps messages only in main memory and not on disk. This is much, much faster than storing the messages to disk. It does however bring a higher risk, because if RabbitMQ crashes, the messages will be lost. Turning on persistency will mean your messages are also saved to disk, and can survive a RabbitMQ crash. But at the same time it makes things much slower. We therefore recommend leaving the "rabbitmq-persistent" option off (set to "false"). Note that if you do turn it on, durable queues should also be turned on. Persistent messages are still lost if the queue is not durable, because the queue no longer exists.
The "rabbitmq-lazy" setting toggles whether queues are declared as lazy queues. Lazy queues ensure that messages are always written to disk. This lowers overall performance, but prevents sudden performance drops once RabbitMQ needs to flush to disk, giving predictable performance. See the RabbitMQ documentation for more information on lazy queues.
MailerQ opens a number of different connections to RabbitMQ, and each connection runs in its own thread. There are separate threads for consuming from the inbox queue and threads for publishing messages to the result queues and/or back to the outbox queue.
You can specify in the config file that you want to start up multiple consumer and/or multiple publisher threads. If you notice that the consumer of publisher threads are CPU bound, you can configure MailerQ to start up more threads.
rabbitmq-consumers: 1 (default: 1)
rabbitmq-publishers: 1 (default: 1)By adding the "rabbitmq-consumers" and "rabbitmq-publishers" variables to the config file, you instruct MailerQ to start up more consumer and/or publisher threads.
The data stream between RabbitMQ and MailerQ can be large. It can even be that big that it takes up a significant portion of the capacity of your internal network. To reduce the load on the network, MailerQ supports gzip compression.
MailerQ normally expects messages from RabbitMQ to be JSON encoded. However, if the AMQP envelope in which the message is wrapped has the "content-encoding" property set to "gzip", MailerQ expects the message to be a gzip compressed JSON object instead. It first decompresses the message, then it parses the JSON. This means that you can publish both normal JSON encoded messages to RabbitMQ, as well as gzip compressed data. If you compress your input JSON, you do have to make sure that you also set the "content-encoding" header in the AMQP envelope that is published to RabbitMQ.
MailerQ not only consumes messages from RabbitMQ, it also publishes messages back to it, like the results that are sent to the result queues or the retries that are published back to the outbox queue. By default, MailerQ only sends pure JSON to RabbitMQ, without compressing it, even if you used compression yourself when you published the message to the outbox. If you want MailerQ to compress the data too, you can use a special setting in the config file.
rabbitmq-encoding: gzipThe above setting only affects how MailerQ publishes messages to RabbitMQ. Messages consumed from RabbitMQ can still both be compressed or not, because MailerQ always inspects the "content-encoding" header from the AMQP envelope.
When consuming from the outbox queue, or flushing a queue, MailerQ loads a certain number of messages at the same time, also called QoS. These numbers can be tweaked to lower memory usage.
rabbitmq-qos: 1000 (default: 1000)
rabbitmq-flush-qos: 1000 (default: 1000)
rabbitmq-minqos: 100 (default: 100)This number is interpreted per consumer thread. Therefore, the maximum total number of messages from the outbox that are in memory at the same time is the amount of consumers multiplied by the maximum number of messages. For example, if "rabbitmq-consumers" is set to 5, and "rabbitmq-qos" is set to 200, the maximum total number of unacknowledged messages from the outbox queue is 1000.
The "rabbitmq-flush-qos" can be set to lower the loaded messages when a flush happens. Although MailerQ doesn't load in the messages, their bodies will be sent over TCP and so the TCP buffer may be fully populated with all this data. As this can happen for anything that is unpaused or resumes sending, a lot may be flushed at once.
"rabbitmq-minqos" is the lower bound for the QoS when MailerQ decides to scale back for performance reasons. The QoS will never go below this number, unless there aren't enought messages to reach it.
To ensure messages are not lost, MailerQ uses publisher confirms by default. This makes sure that messages are not acknowledged before RabbitMQ has accepted responsibility for it. This will inherently tie it to the quality of service settings. However, to further tweak performance, a throttle can be set. This throttle makes sure that only a certain number of messages are actually sent over the socket. The advantage of this is that it reduces the load on RabbitMQ, and keeps the socket open to bidirectional communication. It used to be the case that MailerQ could not adequately respond to flow control by RabbitMQ, eventually causing TCP backpressure, shutting down communication.
rabbitmq-throttle: 100 (default: 100)Retrieve the plugin managed data pointer stored for this connection. If you've made a call to MQ_setConnectionData to associate data with a SMTP connection, you can use this function to get back that data pointer.
The data pointer is completely ignored by MailerQ. This API function is only provided to give plugins the possibility to associate data with a connection.
The data pointer is segmented per plugin, so this function will not return data stored by other plugins. If no data was stored earlier, NULL will be returned.
/**
* Retrieve userdata stored using MQ_setConnectionData
* @param connection the connection in which the data was stored
* @return void*
*/
void *MQ_connectionData(MQ_Connection *connection);
Be aware that if you store data in the connection that you had allocated yourself, it is also your responsibility to deallocate that data before the connection is closed.
This is a tool to convert MailerQ output into the format understood by the httpcaller. It is typically used in a technology stack together with MailerQ and HttpCaller, where this tool is the middleware between these two applications.
MailerQ --> RabbitMQ --> AmqpHttp --> RabbitMQ --> HttpCaller --> Your REST APIAmqpHttp is responsible for converting MailerQ output into the format of HttpCaller. Both applications process JSON messages, but with different field names and a different structure. The input for httpcaller also requires some extra fields, like the URL that is to be called.
The application has the following features:
The application is stored in our APT repository. If you have already enabled our APT repository, you can install it with this command:
sudo apt install mailerq-amqphttpYou typically run this program from the command line. It reads all its input from a config file stored in /etc/copernica/amqphttp.txt.
amqphttpAll options can be supplied in the system wide config file, via environment variables and as command line options. The config file is stored in /etc/copernica/amqphttp.txt. Options from the config file can be overridden by passing them as command line arguments or environment variables. The "rabbitmq-address" setting, for example, can be provided in the following ways:
We use "rabbitmq-address" as an example. All configuration settings can be supplied in a similar manner.
The application reads out the MailerQ result queue from RabbitMQ, converts the messages, and publishes them back to a different queue, where they can be picked up by the httpcaller.
The queue name can also be formatted as "exchange:routingkey". If you use this format, all messages published to the specified exchange with the specified routing key are consumed.
The "filter" option can be used to filter out certain fields from the input, for example because you do not want to send ALL data from the results queue to the HTTP endpoint. The filter option should be a space separated of JSON wildcard pointers with the fields that are filtered out from the messages.
filter: results/*/recipient results/*/envelopeThe above filter ensures that the recipient and envelope properties are filtered out of the results array. The filters are JSON pointers with added wildcard support. See https://datatracker.ietf.org/doc/html/rfc6901 for more information about JSON pointers.
You normally want to setup a full RabbitMQ -> AmqpHttp -> HttpCaller stack. The mailerq-webhook can be used to set this all up.
The mq_initialize() function is an optional function that you could implement in a plugin. When MailerQ starts and has loaded all the plugins, it will check every plugin to find out if this mq_initialize() function is available, and if it is, it will call it.
You can use this function to run your own initialization code.
#include <mailerq.h>
/**
* Initialize the plugin
*/
MQ_EXPORT void mq_initialize() {
// @todo add your initialization code
}The counterpart of the mq_initialize() function is the mq_cleanup() function.
If you want to change the recipient address that is associated with a MQ_Message, you can use this function. This function allows you to store a new recipient address in a message.
/**
* Update the recipient address
*
* @param message the message to set the recipient address in
* @param recipient actual recipient address
* @param size length of the recipient string
*/
void MQ_setRecipient(MQ_Message *message, const char *recipient, size_t size);This function has a different behavior if you call it on a message that is being received and on a message that is being sent. If you call it on a message that is being received, for example inside your mq_smtp_in_message() function, it is simply an alias for setting the "recipient" property in the JSON. Changing the recipient address by calling this function, will also change the JSON that is going to be published to RabbitMQ.
For outgoing messages however, changing the recipient with this function will not modify the JSON data. If your plugin calls this function before the "RCPT TO" instruction is sent, the newly set recipient address is going to be used in the SMTP protocol, but when the message is published back to RabbitMQ (for example to the result queue, or back to the outbox queue for a retry) it will still have the original recipient address in the JSON. If you also want to modify the recipient address in the JSON data, you should use the MQ_json() function as well.
