Release notes
The MailerQ development team is always busy developing
and improving the MailerQ MTA. On this page, you can find the
improvements of MailerQ for every release.
Improvements:
- Removed an undocumented and confusing feature that downloaded a webpage when the text version of a responsive email contained only a URL
Bug fixes:
- Resolved an issue affecting outgoing HTTP requests, particularly when interacting with webservers utilizing chunked transfer encoding
Improvements:
- HTTP(S) servers can now be used a storage backend
- Responsive message can now be text-only
- Download proxy can now dynamically resolve domain names
- DKIM and ARC signatures now always include List-Unsubscribe-Post header
Bug fixes:
- MX Patterns now get deleted when its group gets deleted
Bug fixes:
- Fixed a possible segmentation fault during DNS lookups
- Fixed support for SQLiteQ in Ubuntu 22.04
- Fixed error when enforcing STARTTLS
Improvements:
- Added support for openSSL 3.0
- Error messages about operations in MongoDB have been improved
- Display seperate version numbers for the frontend and backend packages
Improvements:
- External MTA ips are loaded with a single DNS lookup
Bug fixes:
- Fixed error when enforcing STARTTLS
- Rewrite rules now work for the default pool
Improvements:
- Added option "smtp-hostname" to manually specify hostname to return to SMTP clients
- Hostnames can be used for proxies/external MTAs in addition to regular IP addresses
- Added support for specifying IP ranges for proxies/external MTA's public IPs
- Added HAPROXY support for proxies/external MTAs
- Updated AMQP-CPP library to version 4.3.26 (property "connection_name" is now set during connection setup, making the connection stand out more in rabbitmq UI)
Bug fixes:
- Fixed crash in receiving email caused by mails to non-local addresses
- Fixed memory leak when sending via external MTAs
- Fixed disappearing messages from temporary queue being destroyed by another MailerQ instance
- Fixed possible buffer overflow on failed HTTP connections
- Fixed /dev/shm directory fill-up with DMARC reports
Improvements:
- No relevant functional changes (some dependencies on internal libraries were updated)
Improvements:
- Added support for libidn12
Improvements:
- Updated AMQP-CPP library to version 4.3.24 (no functional changes)
Bug fixes:
- Fixed issue with initialization of the DNS resolver which could cause some DNS lookups to fail
Improvements:
- No relevant functional changes
Improvements:
- Added support for chunked data transfer on HTTP requests
Improvements:
- Updated AMQP-CPP library to version 4.3.20 (no functional changes)
Improvements:
- No relevant functional changes (some dependencies on internal libraries were updated)
Improvements:
- Re-added support for Couchbase
- Updated the OpenSSL version
Improvements:
- Improved "timeout loading key from storage" messages (the missing key was not reported)
- It is now possible to create cluster-wide external MTA's
- Added support for libidn 11 and libidn 12
- HTTP downloads: better error messages for failed downloads
- HTTP downloads: added support for http://1.2.3.4 URLs (where the hostname happens to be an IP address)
- Internal webserver: methods other than the standard GET, POST, etc now trigger an error
- Filtered issue with mongodb:// addresses when a "readAttempts" parameter was supplied
Bug fixes:
- Fixed the output of the "--fetch-license" flag (it incorrectly instructed to save the license file as "1" instead of "/etc/mailerq/license.txt")
- Fixed compatibility with RabbitMQ servers without a max frame size
- Fixed race condition when setting up the TCP connection with RabbitMQ
Changes:
- Temporarily removed Couchbase support because the couchbase client API library has changed too much
Improvements:
- Improved memory usage when parsing xml files for DMARC reports
Bug fixes:
- Fixed a memory leak when using openSSL
Improvements:
- The algorithm to detect modified files (for example new SSL certificates) has been improved to also detect changes in symlink targets
- It is now possible to use inside the list-unsubscribe header
Bug fixes:
- Fixed possible buffer overflow when parsing PROXY protocol buffers
- Fixed possible deadlock when expected HAPROXY header is not received
Improvements:
- Added support for command line flag --ignore-global-config to ignore the global /etc/copernica/config.txt file
- Added support for the MAILERQ_IGNORE_GLOBAL_CONFIG environment variable to ignore the /etc/copernica/config.txt file
- Added support for the MAILERQ_CONFIG_FILE environment variable to specify an additional config files to be parsed
- Added support for a "config-file" option to specify from a config file whether an additional config file should be parsed
- The session-cookie for the management console now gets extra settings "SameSite=Strict" and "Secure" property to improve security
- Added "www-samesite" config file setting to override the "Strict" setting for the SameSite setting
- When mailerq is started with "mailerq-4.4", we now also the config file "config-4.4" (the executable name is leading, no longer the built in version number)
- It now is possible to run MailerQ without a valid www directory
- Improved error message when a file or directory does not exist during startup
- Removed a newline from the question whether you want to store the new license file
- The "--config-file" option now overrides the default config file (previously, we always parse the default config file)
- We no longer try to fetch storage keys that are the empty string
- Updated the public suffix list for domain-to-organization mapping
- Memory optimizations in parsing and generating mails
- Updated the supported openssl versions up to number 1.1.1n
- More specific handling for mysql/mariadb connections (mariadb now has slightly different handling than mysql)
Improvements when downloading HTTP resources
- The error messages have been improved
- Fixed handling multiple requests over a single connection for which "transfer-encoding: chunked" was set
- Fixed handling a case of a HEAD requests
- Fixed an issue with "transfer-encoding: chunked" the chunk-size was not correctly parsed for very small chunks
Bug fixes:
- Fixed issue that caused MX records with overlapping IP addresses in MX records to use the wrong MX priority
- Fixed issue with injection via SMTP in combination with storage: the TTL for email bodies was not correctly set, which could lead to email bodies being removed from storage before they message in RabbitMQ expired
- Checking the username of the smarthost contained a bug that causes an error.
- Fixed a bug in Message-ID generation that could result in duplicate Message-IDs
- Fixed possible buffer overflow when adding headers to email messages
- Fixed possible deadlock state in SSL handling
- Fixed issue when command line arguments appeared more than once
Improvements:
- Added logging of failed authentication attempts in application log.
- Reduced the time brute force protection is active on an IP to 15 minutes since last failed attempt.
- Replaced deprecated www-password setting in the default config file with www-auth.
- UI updated MTA dropdown in SMTP monitor, and settings forms.
- Updated AMQP-CPP library to version 4.3.15.
- Temporary DNS errors are now recognized as either timeout or servfail.
Bug fixes:
- Fixed tab character encoding for the Quoted-Printable encoding.
- UI: Added missing "domain" field to paused errors creation form.
- UI: Fixed a possible error when a settings slide is closed too fast.
Improvements:
- Adjusted default DNS settings to improve resolution in case of many non-existing domains.
- Updated AMQP-CPP library version to 4.3.14 (this fixes a number of AMQP related bugs).
- Default IP pool can now be used in 'Set IP pool' rewrite rule.
- Improved performance of loading log files in UI.
- Improved error reporting for AMQPS connection.
- Improved error message for storage lookup timeout.
- A number of small performance improvements.
Bug fixes:
- Fixed encoding errors in quoted printable encoding when using responsive email.
- Fixed flush-operation being possibly destructed from the outside.
- Fixed a possible crash when removing an external mta from the management console.
- Fixed a possible crash happening on log rotation.
- Fixed segfault when constructing temporary queue name for some edge cases.
- Fixed leaking file descriptors when using embedded images with responsive email.
Improvements:
- UI: Updated MTA IP/Pool dropdown in the omniheader.
Bug fixes:
- Fixed a bug in garbage collection on IP Pool level.
Bug fixes:
- Disabled link-time optimization for now as it breaks plugins.
New features:
- Added optional expiration time to Paused Deliveries and Response Pattern action "Pause sending".
- MailerQ now accepts environment variables for config values.
Improvements:
- Enabled link time optimization, which improves performance and reduces binary size.
- Improved performance of DNS resolution.
- Improved performance of domain aliases.
- Improved performance of garbage colletion.
- Improved performance of pause timers.
- OpenSSL security level is now set to 1 explicitly, to allow using TLS with older servers on Ubuntu 20.04.
- Print and log an error message when the inotify watcher is unable to start.
- Retry reverse DNS lookup of MTA IPs for non-temporary error types and log warnings in application log.
- Set default value for "max messages per connection" to 50 in the built-in default email throttle.
- Temporary queues can now also be created for a set of IPs.
- Updated DNS configuration options to match DNS-CPP library. See "Documentation" -> "DNS settings" for more details.
- Upgraded to DNS-CPP-1.2.3 library, which solves a lot of DNS resolution related bugs since MailerQ 5.12.0.
- UI: Added check for valid JSON pointer in Rewrite Rule conditions.
- UI: Added warning when submitting empty form to REST API.
- UI: Sessions are now reloaded from the database when MailerQ restarts.
- UI: Display logged in user, license information and limitations in the UI.
- UI: Removed unused "Quality-of-service for temprorary queues" from email throttles.
Bug fixes:
- Fixed a possible crash in AMQP connection.
- Fixed a possible crash in cleanup when using directory message store.
- Fixed a possible crash in websocket monitor.
- Fixed a possible crash on invalid Content-ID header when using responsive email.
- Fixed a possible crash when evaluating SPF record without a redirect.
- Fixed a possible crash when modifying web session data.
- Fixed a possible crash when sending messages using an unavailable External MTA IP.
- Fixed a possible crash when using embedded images.
- Fixed a possible crash with uncaught exception when trying to reconnect to the cluster exchange.
- Fixed a possilbe crash when a tag is set to null.
- Fixed a possible race condition and deadlock when using responsive email.
- Fixed application hanging in when a certain RabbitMQ connection or channel error occurs.
- Fixed application hanging When declaring the queue for the cluster connection fails.
- Fixed application hanging on shutdown.
- Fixed DNS resolver not picking up system settings correctly.
- Fixed double timestamp when logging to syslog.
- Fixed empty IP Pools getting removed from memory.
- Fixed handling of loopback IP.
- Fixed incorrect domaingroup size counter.
- Fixed incorrect handling of Rewrite Rule action "smarthost" with IP address.
- Fixed issue with DirWatchers which could cause the webserver to cause MailerQ to freeze on shutdown.
- Fixed issue with slow DNS resolvers resulting in excessive temporary DNS failures.
- Fixed memory use slowly growing over time due to issues in garbage collection.
- Fixed offline MTA IPs not getting cleaned up.
- Fixed sending empty paused events without ID.
- Fixed TLS certificate verification for amqps:// connections.
- Fixed user permissions for directory message store.
- Fixed using private domains (i.e. ".internal") with licenses that have domain limits.
- Long queue names should be properly truncated to 255 byte limit.
- Removed unnecessary KeyUsage check for "verified" connection security.
- UI: Fixed cluster information showing up at the bottom of the UI instead of in the menu.
- UI: Fixed filter in Delivery intercepts page.
- UI: Fixed forced error toggle.
- UI: Fixed items flashing in drag and drop lists.
- UI: Fixed missing title and description on "Flood Patterns" settings page.
- UI: Fixed search on "name" for "Delivery Intercepts" slide.
- UI: Fixed SMTP Connections widget, where counters were going outside the parent container.
- UI: Fixed UI hanging if left in inactive tab for a longer time with a lot of activity.
- UI: Fixed pausing on MTA IP level in UI not working.
New features:
- Multi-user authentication for SMTP injection and web interface. Supported backends are: LDAP, RADIUS, Linux users, HTTP endpoints, username/password files and custom scripts. For more detials see:
SMTP server documentation and Management Console documentation
- Web interface has named sessions now, and username is available in www-logs.
Improvements:
- Significant performance increase (up to 8x) in DNS resolution.
- Secure connection can now be set on the message level.
- MTA IPs are now paused if their PTR lookup failed with some transient issue, temporarily parking messages.
- DNS resolve expire time, spread, interval and buffer size are now configurable.
Bug fixes:
- Fixed sporadic failure on startup (interrupted system call).
- If DNS subrequest fails during MX IP lookup, we mark the error as temporary instead of nodata.
- Fixed connections being made to servers not supporting TLS when TLS is required, when the information about missing STARTTLS is cached in memory.
- Fixed issue with External MTA IPs without PTR records.
- Fixed use of uninitialized value causing unpredictable results when using external MTAs when reverse names were not properly set up.
- Fixed proxy handshake failure potentially causing application crash.
- Channel error was incorrectly interpreted as 'name' error even though the connection was no longer usable, causing a SIGABRT.
- Fixed queue churn on domain group split because not all reduced properties where properly installed everywhere.
- On transient DNS issues, MTA IPs retry their name resolution much more quickly.
- Fixed missing error counter increment when first connecting to an mta and we conclude the security is insufficient.
- Fixed a bug causing space to changed to %20 in responsive email attachment names.
- Made sure everything has reasonable lower bounds in the config file.
- UI: Fixed Response Patterns list not closing.
Improvements:
- S3 storage: Show AWS error message on startup failure.
- Improved error messages on spamassassin, smtp-sink and proxy connection failure.
Bug fixes:
- Fixed a possible segfault in smtp thread.
- Fixed localhost resolving issue when no local DSN recursor is present.
- Fixed incorrect handling of some errors in "intro" state. Greylisted MTAs should not add attempts to messages, but should leave the messages to be processed later.
Known issues:
- If there's no local DNS recursor, "localhost" addresses in the configuration file may not be recognized and should be replaced with IP address 127.0.0.1. This will be addressed shortly in a bugfix release.
New features:
- New Rewrite Rules actions allowing JSON manipulation: "Set JSON property", "Unset JSON property" and "Apply JSON patch".
- Replaced synchronous DNS library (libresolv) with asynchronous DNS-CPP.
- Enabled RabbitMQ publisher confirms (new config file setting: rabbitmq-throttle with default value of 100). This prevents runaway backpressure on the outbox queue when under heavy load.
- Quality-of-service for temporary queues is now dynamically determined based on queue speed and limits.
- Added support for content-id in Responsive Email templates. Added "embedded" property, which is similar to "attachments", but has additional "id" property for setting content-id.
Improvements:
- Improved speed of database operations (web UI settings like pause/resume)
- IP Pool is now automatically paused (with an issue notification in UI) if there are IPs defined in the database, but not present on the machine.
- Removed algorithm that picked the least unique message from the fornt of the outbox queue, it decreased performance for marginal scheduling improvements.
- No longer reload the database settings when no changes took place in a REST call (i.e. incorrect call).
- Do not clear all temporary/parked queues when pausing on the root level.
- Application is now paused instead of killed on license resolution failure.
- License server connection failure sends notification to the user via web-push.
- Flood pattern is now also moved to a new domain group on a domain split (change of MX records).
- Web UI sessions no longer depend on the Client IP to allow for connections through a loadbalancer.
- Removed unnecessary SPF checks on MTA IPs.
- Deprecated the no longer used database-threads option.
- Deprecated the no longer used dns-threads and dns-getaddrinfo options.
Bug fixes:
- Fixed hang on S3 message store connections.
- Fixed possible crash on incorrect MIME comments.
- Fixed encoding of UTF8 attachment names, it is now compliant with RFC2231.
- Make sure log downloader doesn't start in stopped state.
- UI: fixed "explain throttle" widget jumping to built-in without user action.
- UI: fixed "explain throttle" widget showing incorrect max new connections number.
- IP Pool names are now immutable to prevent runtime errors.
- Fixed thread race with certificate/cipher for secure connection.
- Fixed Email Throttle option "Security" set to "encrypted" or "secure" on a server without starttls banner causing infinite loop and reconnects.
- Fixed incorrect memory free on shutdown.
- Fixed a crash when the reverse DNS lookup for an external MTA failed.
- Do not drop additional database reload requests while we're reloading but instead queue them.
- When removing an IP from a network interface card, we were doing too many reverse DNS lookups.
- Ficed handling exceptions when constructing an Email::Message.
Improvements:
- UI: Add debouncing (delay between input and action) to list filters.
Bug fixes:
- UI: Fixed web interface not working over HTTP. (Don't register a service worker when the functionality is not available in the browser)
- Tags were not properly cleaned up on mta to remote ip level.
- UI: Fixed limit message not rendering in table widgets with limited rows.
New features:
- New Response Pattern action - "Add email address to Suppression List".
Improvements:
- In-memory queue limit cannot exceed a minute of messages at max, allowing for better resource allocation.
- DKIM keys now default to SHA256 instead of SHA1 if there's no 'h=' field in the DKIM DNS record.
- Email preview module can now be disabled with "render-enabled: false" setting.
- External MTA IP info is now also written to ips table in database.
- Temporary queues in RabbitMQ now use IP Pool's name instead of its ID.
- SMTP and Spool injector now contain /extracted/{from,to,subject,recipient,envelope} keys, like the AMQP injector.
- Servers with misbehaving TLS get marked for 5 minutes and no TLS will happen to those (despite advertising STARTTLS).
- UI: Supprssion List view is limited to 100 items and has a filter bar.
- UI: Implemented endpoints for web push notifications.
Bug fixes:
- Fixed setting properties or applying json patches in response patterns potentially causing application crash.
- Fixed infinite cycling and pumping around of message on startup if mta-domaingroup or pool-domaingroup was paused (happened on low number of paused messages)
- Fixed messages not being released from in-memory queue when the queue was paused.
- Fixed temporary queue not always cleaned up after it was done and empty, depending on how the last message got removed.
- Fixed unbounded memory queue during resolution of domain. It is capped to default capacity now.
- Prevent deep queues from being created on DNS error, reducing resource usage.
- Fixed uncaught exception (bad file descriptor) in the publisher thread could forcefully quit application.
- Fixed missing queue counters on various levels for the newly added deep queues could cause large discrepancy between (rabbit) loaded and in-memory.
- Fixed outbox QoS potentially growing during application runtime.
- Fixed tags not properly counted on in-memory queues on mta-domain groups and pool-domain groups.
- When dns-threads is set to 0 in the config, report an error and stop.
- Fixed incorect handling of "final" setting when disabling/enabling rewrite rules.
- Fixed handling of temporary dns failures, the 4xx error codes are now temporary failure.
- Fixed configuration file update script.
- UI: Fix missing events on IPPool level.
Improvements:
- Allow incoming SMTP connection brute-force protection to be disabled in the config/cli (smtp-protection: false).
- REST API: add support for HTTP and Socks proxy in External MTAs.
- Improved performance of data processing in the web interface.
- Forced failures by Rewrite Rules are now counted in the web interface.
Bug fixes:
- Log downloader threads are now shut down when MailerQ shuts down (prevents segfault).
- Fixed inverted suppressions type, addresses were returned as type domain and vice versa.
- Fixed potential hang on tempqueues, they now remove everything from the loop on channel error (allows publisher thread to terminate).
- Fixed crash in shutdown when rabbit is closed (publisher could lock destructed mutex).
- When MailerQ shuts down due to rabbit/cluster/mongo connection lost, the returned status code should be failure.
- Only the first forced error for a domaingroup, pooldomaingroup or mtadomaingroup was picked up, even if it was a tagged one.
- Pool/MTA domain groups could be destructed too early which caused some events not to be sent out.
- Forced error now sending new event on edit of cluster event as well + unblocked now also has alias info.
- Enforce the pooldomaingroup correctly into memory if an error is specified on it.
- Fixed a bug with a crash on missing database client library, a proper error message is now logged.
- UI: Respond to error in MTA name edit form.
- UI: MTA name is updated correctly on change.
- UI: Use proper names instead of internal names for dashboard widgets in Add Widget menu.
New features:
- HTTP proxy and SOCKS proxy support for External MTA IPs
- IP range grouping
Performance:
- QoS and queue size are now applied on IP pool-domain and MTA IP-domain level (used to be domain level only).
- Outbox queue QoS is now dynamically scaled back to rabbitmq-minqos (default: 100) to lower memory usage when many temporary queues are consuming.
- Prevent queue churning. In some cases messages were picked up from the temporary queue just to be placed back in it.
Improvements:
- SMTP timeout for receiving email is now configurable. The new default value is 5 minutes.
- Show openssl errors in starttls when possible.
- Added time filter to log viewer.
- Added 'domain' and 'local' fields to extracted properties for envelope and recipient domain names.
- Added date to log viewer next to the time.
- UI: improved loks of External MTA IPs list.
- UI: show protocol in external MTAs list.
- If a terminal is attached to stdout, MailerQ will now also log there.
Bug fixes:
- Fixed limit of 1 message in-memory would soft-lock queue by setting qos to 0.
- Fixed maxattempt property not picked up correctly from response pattern.
- Fixed root level pause not being picked up on application startup correctly.
- Fixed bug when all MTAs on machine have their name set in database, consumers would not start.
- Fixed issues with queue declarion when queues are already declared with different parameters in RabbitMQ. MailerQ now checks if the queues are correctly created, and gives an error otherwise.
- Fixed logging of database validation errors and hardware id/license errors.
- UI: fixed showing live log instead of send.bin log file in log viewer.
- UI: fixed delivery intercepts not showing default pool correctly.
- UI: refresh MTA dropdown when MTA has been renamed.
- UI: validate MTA name before and after interacting with the database.
- UI: fix MTA name not being reset to DNS resolved value after clearing the entry.
- UI: Fixed Log Viewer filter not working correctly.
- UI: Use JSON pointer instead of dot notation to match backend.
- UI: Make IP pools slider dropdown drop down.
New features:
- Rest API Suppressions endpoint
- Rest API IP Pools endpoint
- Rest API External MTA IPs endpoint
Improvements:
- Result messages now contain message size (/sent/data) and number of bytes sent over a connection (/sent/connection).
- Result messages now contain "/seen" property, which contains timestamp of first time MailerQ consumed the message.
- Only interrupt main thread for post and delete calls when needed.
Bug fixes:
- Fixed crash on startup with empty user config value.
- Fixed a possible crash with 'bad file descriptor' error in publisher thread.
- Fixed potential race condition in smtp thread.
- Fixed External MTA IPs ports higher than 32767 not working correctly.
- Fixed incorrect cleaning of ips in results messages.
- Fixed service file in rpm package. Fixed legacy init scripts.
- Fixed excessive memory use on very uniform binary log files.
- Fixed alias dropdown acting weird on Safari.
- UI: Fixed Web UI not working with newer versions of libmagic (5.35+, Debian 10).
- UI: Fixed rendering on input fields in email throttle page.
- UI: Fixed "messages to load into in-memory queue" setting for flood patterns.
- UI: Fixed small rendering issues on email throttle page.
- UI: Fixed smtp error code validation in suppression list settings page.
- UI: 0.0.0.0 is no longer accepted as a valid external mta public IP.
Bug fixes:
- Fixed rewrite rules not being applied to emails from temporary queues.
- Pause and error events are now correctly handled on domain group splits, fixing mailerq sometimes sending even though being paused.
- Fixed fail and delay actions on simple rewrite rules.
- Fixed MTA IPs dropdown not showing data if the page is opened in the background tab.
- Fixed counters becoming empty on domain group split when filtering on a combination of MTA IP/IP Pool and domain.
- Email trottle, pause and error events are now properly sent when looking at a pool-domain combination and they are updated.
- Improved GUI performance of DKIM keys settings page with large number of keys.
- Limited DKIM keys list to show only the first 100 keys matching the filter.
- Paused deliveries list now shows default pool correctly.
- Forced error list and edit page displays local IP correctly.
- Application no longer waits for DNS resolver on shutdown.
Bug fixes:
- Fixed a bug where the end sequence ("\r\n\.\r\n") in incoming SMTP connection could be not recognized correctly, causing the connections to timeout.
- Fixed IP removal not triggering an update to the cluster.
- Fixed crash on 'failed to create channel: connection not active'.
- Fixed domain filter in rewrite rules ignoring the domain setting.
- Fixed crash during startup if the cluster object was prematurely destructed.
- Emails with smarthost should not be able to go over NAT IPs unless the smarthost port is 25.
Improvements:
- Many small performance improvements due to replacing many unnecessary copy operations with move operations.
- ResponsiveEmail engine no longer generates empty text version for html-only email.
Bug fixes:
- Many small bugfixes found due to improved static code analysis.
- Fixed a possible segfault in cluster communication.
- Fixed a possible segfault when punyencode used uninitialized memory if libidn was not valid.
- Fixed data race in responsive thread initialization, now thread is started after all members have been initialized.
- UI bugfix: Last error widgets displays links to pause, create capacities, and create floodpattern based on logs. These work now.
- Rewrite rule fail actions are no longer implicitly final.
- Fixed a bug causing download cache to not be used correctly.
- Fixed a memory leak in spamassassin integration.
- Fixed a potential memory leak then calling move assignment on existing document.
Improvements:
- Simple rewrite rules are executed in the worker threads instead of the main thread.
- Performance optimization in the logger.
Bug fixes:
- Fixed a bug where the end sequence ("\r\n\.\r\n") in incoming SMTP connection could be not recognized correctly, causing the connections to timeout.
- Fixed a bug with MTA names loaded from the database, which were lost within an hour, falling back to PTR records.
- Fixed external MTA IPs not working correctly when an MTA name was specified.
- Fixed spamassassin check not working correctly if DMARC check wasn't also enabled.
- Fixed incorrectly copied IPv6 address when connecting to a remote endpoint.
- Fixed a bug where MTAs no longer present on the machine could still be used by the pool.
- Fixed a bug where server id was sometimes incorrect.
- Fixed incorrect bind IP when using smtp-sink with External MTA IPs.
- Fixed a possible segfault on an ivalid recipient.
- Fixed a possible segfault when MTA IPs are added or removed from a pool.
- Fixed a possible race condition in log file monitor.
- Fixed a possible race condition during database updates.
- Fixed a possible data race in application initialization.
- Fixed a race condition in the logger.
- Fixed typos in UI.
New features:
- Email address and domain name suppression list
- External MTA IPs support - used for public IPs behind NAT (i.e. in a cloud environment)
Improvements:
- Forced errors, rewrite rules, DKIM keys and REST tokens can now be non destructively disabled.
- MTA IPs can now be added / removed during runtime and will be dynamically updated.
- Performance: opportunistically move the messages to deeper queues to get more, shorter queues on IP->domain level.
- Rewrite Rules can now fail and delay messages before any attempts are actually made.
- MTA IPs initialization is no longer blockingly resolving the ptr records, they are paused until the hostname is resolved for the first time.
- Refactor of loading MTA IPs. Consumers are now started only when MTA IPs are fully loaded.
- Log file viewer now also supports non standard names for binary logs.
- Raised license ping timeout from 10s to 60s to mitigate sporadic failures.
- Cluster connection tries to restart for 60 seconds, if an error occurs, to prevent mailerq from exiting on transient errors.
- Garbage collection will be paused if the database is reloaded during cleanup, and will be resumed later.
- Flood patterns match type now defaults to substring in the UI.
- UI: Active throttle explain widget now correctly shows settings that are not applicable on some levels.
Bug fixes:
- Fixed data-race in std::shared_ptr usage. This might have cause a number of segmentation faults.
- Fixed skewed SMTP thread load balancing.
- Fixed race condition in MTA IP info.
- Fixed segfault on delaying a message with missing/invalid recipient.
- Fixed layout of many forms (rest token, ip pool, rewrite rule, etc) to properly show a scroll when in a short window (small height).
- Fixed bug in rewrite rules list where changing conditions would not update the list.
- Fixed a possible segfault when running multiple plugins.
- Fixed application hanging on the web thread on failed startup
- Fixed handling of IP Pool id in email throttle form.
- Fixed POST /pattern endpoint to match properties from GET /patterns (this fixes UI bug with setting flood pattern type).
- Fixed displaying the cascaded security setting incorrectly exposed as 'opportunistic' in the active throttle widget.
- Fixed possible XSS in web interface notifications.
- Fixed incorrect fallback to A record if there are MX records present without any records.
- Fixed memory leak in download operation.
Improvements:
- Email throttles now have set precedence and can be reordered using drag&drop. Only the first matching throttle on a given level is applied. This prevents issues with conflicting throttles.
- Email throttles and flood pattern edit pages have been simplified. The least used settings are now hidden behind a 'show advanced' button.
- In app documentation in email throttles and flood pattern edit pages has been improved.
- Application has been updated to use the latest AMQP-CPP version (4.1.5) for communication with RabbitMQ.
- Message-id is now generated for every message to allow deterministic message tracking.
Performance:
- Web interface has been moved to its own thread.
- Get handlers and websockets now only interrupt when they need to, most calls completely bypass main thread.
- Instructions interrupting main thread are now moved instead of copied.
Bug fixes:
- Fixed a bug which could cause bad file descriptor exception.
- Fixed segfault if no global authentication username/password was set, but a plugin took control over it.
- Fixed a bug where Mailerq could only load single plugin without segfaulting.
- Fixed crash on failing to open the binary log file during runtime.
- Fixed misalignment issue with conditions and actions in the rewrite rules list UI.
- Fixed handling of 0 and negative values in email throttles and flood patterns UI.
- Fixed a number of small issues in web interface.
- Fixed wrong permissions of the logs directory which prevented logrotate from rotating the logs.
- Fixed default values in the configuration file generator script.
Improvements:
- Rewrite rules are now idempotent. They actions are only applied in-memory, unless we're publishing to result queues.
- Plugins can now write to the application log via MQ_log(priority, message) function. (version not bumped, ABI compatible)
- Combined logfiles now also correctly compare against symlinks, opening a single file instead of the same file twice.
- If multiple settings refer to the same binary log, output an error and fail to start
Bug fixes:
- Fixed a bug where default SMTP IPs were incorrectly included when IP pool is set on SMTP injection.
- Fixed pipe usability check not taking pool into account.
- Fixed initial burst of messages on very low email throttles.
- Fixed incorrect server identifier casting in the REST API, that resulted in incorrect queries.
- Fixed last element of log being displayed because socket was closed before data was sent.
- Fixed a crash on broken templates when personalizing an in-memory responsive document.
- Fixed default IPs overriding the IP pool.
New features:
- REST API: Added forced errors endpoint
Improvements:
- Extracted headers are now canonicalized to lowercase for logging in received-log-format and send-log-format. For example "message-id", "Message-Id" and "Message-ID" can all be logged with "{$email.extracted.headers.message_id}".
- Improved performance of the counters in the main thread
- Added support for username login in Couchbase connection string
- Added support for custom ports for database-address
Bug fixes:
- Fixed editing of the Rewrite Rules actions like "Set IP"
- Fixed the attempt number not incremented on uconfirmed delivery attempts (timeout after the data was sent).
- Fixed pause delivery form not sending data.
- Fixed possible segfault on envelope equals and envelope contains
- Fixed republishing of messages when instance is started in paused state (high CPU usage).
- Fixed hang on incorrect connection string for Couchbase
New features:
This release introduces first endpoints of the REST API. For the documentation, please visit REST-API-v1 documentation page
- REST API: Added "REST API tokens" management settings page
- REST API: Added message injection endpoint
- REST API: Added pauses endpoint
- Added new Rewrite Rules conditions: json-property-matches, json-property-not-contains.
- Added new Response Actions "patch" and "patchresult", allowing RFC 6902 JSON patches.
Management console:
- Refactored the UI code.
- It is now possible to use URL to link to the particular settings page.
- Improved in-app documentation.
- Renamed Rewrite Rule condition's "Property" input to "Property pointer" and explained to use JSON Pointer inside.
Improvements:
- Reduce queue flushing of paused messages on domain group split.
- www-log now also contains the authentication used ('none', 'cookie', or rest token name)
- Improved performance of garbage collector
Bug fixes:
- "Subject equals" Rewrite Rule condition could segfault if no subject was set.
- Rewrite actions without a trigger are no longer accepted.
- UI bugfix: Prevent HTTP 500 error on incorrect email address/regex in "Local email addresses" setting.
- UI bugfix: Fixed rewrites list not updating after an edit.
- UI bugfix: Fixed edit smarthost action in the rewrite rules.
- UI bugfix: Fixed handling of closing slides with ESC key.
- UI bugfix: Fixed rewrites rules forms closing when ajax request failed.
Improvements:
- Headers are now also extracted on incoming SMTP emails.
- Added support for JSON pointers in Rewrite Rules JSON property conditions.
- Imrpoved the drag and drop interface in Rewrite Rules list.
Bug fixes:
- Fixed bug where set IP Pool action form would create an action when no IP Pool was selected.
- Fixed bug when the action would reside on the list after it was removed.
- Fixed pause button not working in the live log.
- Fixed bug with restore dashboard functionality.
- Fixed logrotate config file.
- Fixed default config values on upgrade.
Bug fixes:
- Fixed a bug where --fetch-license flag failed to take proxy setting into account
- Fixed a bug where connection counter would not be lowered if scheduled connections were cancelled
New Features:
- HTTP and SOCKS5 proxy support.
- Log rotation is no longer handled by MailerQ, but by logrotate service.
- Filenames in the logfiles can now be the same and will be handled correctly as a single, logical file.
- New config file options `strict-email-address`, `validate-envelope`, and `validate-recipient` added to allow loose email address parsing, e.g. @
- Support for different DKIM header/body canonicalization types
Management console:
- Improved Rewrite Rules list page with additional information.
- Improved the filter in log viewer.
- Improved data loading indicators in web interface widgets.
- Last errors widget can now be refreshed.
- Fixed ordering of Rewrite Rules in web interface.
Improvements:
- Rewrite Rules can now be named.
- Added Rewrite Rule condition "property not equals"
- Simplified logging options, now there is only '*-log: ' and '*-bin-log: '
- Performance improvement - parked queues are flushed less often
- Introduced `smtp-connections-out` and `smtp-connections-in` config file options to specify the maximum number of incoming and outgoing connections separately.
Bug fixes:
- Fixed support for MySQL 5.7.
- Fixed MySQL support when MyISAM engine is not available.
- Messages that reached its last attempt are no longer republished, just to expire on the next attempt.
- Fixed small typo that make the tags table to look like it is stuck.
- Queues generated for local addresses now also take queue properties like max priority & lazy queues into account
- If priority queues are enabled, all queues that will be published to (except delayed outbox queues) will now use priority queues.
- If priority queues are enabled and inbox / other email queues are missing, mailerq would hang.
- If the cluster and rabbit exchange are the same, mailerq will no longer start and give an error instead
- Fixed some rendering issues in Safari browser.
- Fixed segfault in logreader thread when reading out empty logfiles
- Only the last header was stored in `extracted` json field, should have been _all_ headers
- Fixed issue when changing result filter in the LogFilter would not change the log feed.
- Fixed issue when a Rewrite Rules list would not be able to scroll when there's many rules.
- Fixed a possible segfault when loading plugins.
- Fixed bug when responsive html was created even though raw html was supplied, causing two HTML blocks in a single email
- MailerQ no longer throws if any part of the arc chain is invalid
- MailerQ now only parses dkim signatures/arc chain if signing is required
New Features:
- Support for Rewrite Rule conditions based on message properties.
- New rewrite rules actions: set IP Pool, unset IP Pool.
- Support for DKIM oversigning.
Management console:
When upgrading from an older version of MailerQ, we recommend clicking the restore dashboard button at the bottom of the management console, to see all the new widgets.
- Updated the management console with new widgets:
- upcoming retries time distribution
- top domains in the retry queues
- top tags in the retry queues
- last errors
- Improved the performance of management console when under high load.
- Added a total queued column in the table widgets, which is the total sum of in-memory, assigned and parked messages.
- Success and failure rates now show a single decimal point precision.
- Improved the documentation inside the web interface.
- Dashboard restore clears now also clears the widget config.
- The web interface shows visual errors when a query fails.
- Simplified the start up procedure of the UI.
- Force an update of the interface if it has been stuck for longer than 5 seconds due to high load.
Improvements:
- SMTP decoding for the most common case is about 50x faster
- The limit for max characters in allowed names for throttles, schedules and patterns is raised from 40 to 200.
- Process rewrites after email validation to allow rewrite conditions based on MIME content
- MailerQ now logs the number of in-memory objects every minute (number of domains, remote IPs etc.)
- When a new rewrite is added, messages that are in memory / temporary queues are now evaluated again immediately
- Expose next retry time per domain on MTA IP level when the offline event occurs
- Raised default QoS for temporary queues to 10 (1 is way too low)
- Potentially bypass 16, 8, 4, 2, and 1 queue
- Improve message counter accuracy (tags were not always counted).
Bug fixes:
- Fixed performance issue with garbage collection.
- Fixed memory leak in domain groups garbage collection.
- Dns ttl now has a minimum of 60 seconds by default and is configurable through * dns-minimum-ttl setting (fixes very-short lived or even 0 TTLs for MX records)
- IP DNS record TTL for smarthosts are now properly respected
- Using a JSON accessor with more than one level caused issues with the original json. It affected response patterns and rewrite rules dealing with json properties.
- The preferred delivery time was incorrectly published (initial schedule) instead of the newly written one
- Timestamp and priority were incorrectly set during publishing (regression in 5.4.0)
- Smarthost port was incorrectly propagated as connection port, even if no smarthost was set up
- Emails with non-existing pool were not marked as unassignable
- Fixed bug when the pattern type would not be correctly selected on the response pattern trigger page.
- Fixed errors counter to include unassignable error.
- Fixed rendering issue with TagsPicker.
- Delayed queues should also be lazy, when rabbitmq-lazy is set.
- Fixed bug when some objects were incorrectly flushed and messages (and groups) could remain in memory
- Fixed a possible segfault in the rewrite trigger when pool was set to nullptr.
- Fixed the rewrite rule list item to properly handle IP pools.
- Fixed the rewrite rule trigger form to properly handle IP pools.
- Fixed a possible hang on shutdown when using message store.
- Fixed a segfault in download cache when a request was redirected
- Fixed data race in random MIME boundary.
- Fixed the Dns::Hostname class to prevent possible memory leaks.
New Features:
- IP Pools
- MX pattern grouping
Performance optimizations:
- The spool directory now processes files concurrently and asynchronously. The config options `spool-threads` and `spool-open-files` are introduced
- Flush operations are now much faster (approximately 10x) and use significantly less memory
- Vastly reduced allocations during SSL connections, significantly improving performance and lowering resources for secure connections
- MIME encoding for SMTP is now 7x faster
Enhancements:
Management console:
- The queues table contains links to overviews that the queue is related to.
- Added icons and improved formatting in the create response pattern form.
- Grouped settings by category
- Clicking a link the dashboard tables always adds parameters to the filter
New response pattern actions:
- add/remove tags
- add/remove IPs
- set/remove IP pool
New rewrite rules actions:
- add/remove tags
- set/remove IP pool
Spool directory:
- Add option to move files from the spool directory if a failure occurs. The destination directory can be set with the configuration option `spool-fail-directory`
- Check permissions for the spool directory when parsing the configuration
- Lock spool directory files to allow multiple MailerQ instances reading from the same spool directory.
Application log:
- Add default configuration for the application-log: stdout, stderr, syslog, /var/log/mailerq/mailerq.log. Timestamps use local time instead of UTC
- Error-log is now no longer supported, deprecated by the new application-log which is more versatile
- Runtime errors during web requests are now logged instead of silently failing
- Log when SMTP connections are paused/resumed
Other:
- On startup, check if the SMTP certificate and private key are valid and match.
- Check certificate and private key when secure connections to the management console are configured.
- Automatically generate and upload a bug report with the stack trace also generate stack trace on abort
- RabbitMQ configuration: deprecate `max-messages` in favor of `rabbitmq-qos`
- Allow the QoS for flushing to be set using `rabbitmq-flush-qos`
- Allow lazy queue mode to be set by default in the configuration file with option `rabbitmq-lazy`
- Better cache handling for the web interface
- Improved statistics
Bug fixes
- SMTP injection could cause a segfault due to a race condition
- Runtime counter for consumed messages always displayed zero because the seen flag was incorrectly handled
- Empty passwords in smarthost settings were stored in the database incorrectly; if username and password is empty, do not use them to authenticate
- Mail was still being sent despite it being paused on ip-level (pause was not respected)
- Removed tag selection from response patterns; they only worked on the first message.
- Respect local proxy connections
- Fix incorrect text and blank select fields when a rewrite rule is set to end the execution
- Domains are now properly forced into memory, and stay there
- Start processing files in the spool directory only after the user switch, and properly stop during application shutdown
- Fixed integer underflow in hostname size, bounds were incorrectly checked
- Hostnames are now also correctly verified by the web api for capacity and schedule rules
- Temporary queues are now also durable and support priorities
- Message priority was dropped after first consumption if not set in json
- RabbitMQ queues were being redeclared non-durable when they should be durable
- Forced errors were not showing up on the MTA log monitor, and were also not working with filtering the log back on the MTA level
- MTA IPs did not properly evaluate pauses on tags
- Fixed an issue where messages could remain unacked if no more messages were sent
- Fixed a ‘jump’ issue in the interface, where 60 update frames would be sent all at once on an update, pushing out the old data
- Fixed a responsive issue where an incorrect template could cause a segfault
- Improved database upgrade process and stability
- Fixed an issue where downloads could hang if the remote server did not respond
- Fixed potential infinite loop issue during network connection
- Fixed an issue where a hardware id could not be properly generated
- Fixed a possible segfault when DKIM signing an email during canonicalization
- Crashes/uncaught exceptions on threads no longer silently fail
- Fixed a memory leak during the processing of templated emails and user formatted logs
- On a limited license, the messages per minute was incorrectly handled
- Failing home calls are now properly logged
- Emails are now properly rerouted during a split, to be sent from the new group
- UI change: Fixed a growing list of aliases when the domain group splits
- UI change: Made sure that aliases list doesn't overflow
- UI change: Removed grammar and spelling mistakes from management console
- UI change: Fixed description in Paused Form
- UI change: Fixed dropdowns overflow behavior
- UI change: Fixed not detecting paused domains by an alias
- UI change: Fixed style for active throttle widget
- UI change: the Dropdown can render the list on top of the label if it thinks that there is not enough space.
- Packages: Added dependencies on coreutils, liblz4 and libidn11
Bug fixes:
- Fixed bug in personalization of deferred messages when using message store
- Fixed pausing on IP level
Improvements:
- Simplified response patterns. It is now possible to simply set an response pattern action to retry or fail an email, overriding the error code.
- Spool directory pickup is now 3-5x faster.
- SMTP responses are now trated in a case-insensitive manner, in compliance with the standard
- Graylisting decision is no longer based on results-array but on next attempt. Delayed property is no longer used.
- Smtp-sink-address accepts both IPs and hostnames/FQDNs Smtp-sink-ip option is deprecated, smtp-sink-address accepts, IP/FQDN/hostnames. DNS resolution happens on startup. MailerQ fails to start on incorrect value.
Bug fixes:
- Fixed segmentation fault in JsonWrapper.
- Fixed segmentation fault with DSN enabled and missing/malformed address.
- Fixed a bug where MailerQ crashed if expire field was missing for a DKIM key in JSON.
- Fixed a bug causing high cpu usage in main thread.
- Fixed web interface memory leak. SeriesProcessor data rotation bug caused the browser to crash by running out of memory
- Fixed web interface performance issues.
New features:
- Email throttle schedules
- Runtime statistics widget in the management console. (If you're upgrading from previous version, you might need to click on the "restore dashboard" button to activate it.)
Other changes:
- Flood pattern activations are now logged in the errors.log
New features:
- Response patterns
- Flexible, user-formatted log files are now supported using SMART-TPL
Bug fixes:
- Owner for the errors.log is set to mailerq user
- Webviewer can filter logfiles again
- UI: The create throttle rule form allows for empty domains for matching all domains.
- IpAddress was not properly checked on post or pause when it should
- Prevent MailerQ from possibly crashing when Dns::Organization throws
- Responsive emails are now correctly personalized.
- Fix possible int32 overflow
- Messages were stuck in memory if there was no storage attached but the 'key' was set
- No longer empty domain names in wen interface
- MX servers need to be treated case insensitive (hash mismatched)
- Possible segfault on garbage collection of objects, incorrect order / too fast
Other changes:
- Logfiles now output a dash '-' on missing values
- UI: The list of capacities also show the number of patterns assigned to a capacity.
- UI: The list of capacities show '---' when capacity has no name.
- UI: Improved search wording on the list of capacities.
- UI: The list of the remote IPs properly rotate the IPs.
- UI: The list of the remote IPs shows the TTL of the IP.
- Automatically try to fetch a new license once the hardware ID has changed.
- Store logs in a binary file format for internal use, if enabled.
- Logfiles from previous versions are no longer displayed in the web interface.
- Make UA string configurable with new config property download-user-agent
- Send out UA string with all requests
- Fix to the UI: DKIM keys that have an "auto-detect" policy were sending the wrong value to the API
- Fixed possible crash when a DKIM key in the database was not correct, now it just logs the error and ignores the key
- Fixed issue in CLI mode: the application did not close down after mail submission
- Fixed parsing rabbitmq addresses with a comma in the password
- Fixed a bug causing MailerQ to (sometimes) see the license as expired
- Fixed crash when an invalid DKIM key was stored in the database
- Change to the UI: Fixed styles and js code of "Unreachable servers" component
- The www-ip and smtp config file variables can now be used to specify multiple IP addresses to let MailerQ listen to multiple IP's (and possible IPv4 and IPv6 at the same time)
New features:
- Added support for secure amqps:// connections to RabbitMQ
- Added support for "rabbitmq-verify" and "cluster-verify" config file variables that can be used to disable checking the validity of certificates issues by RabbitMQ (usefli> for "amqps://" connections with self-signed certificates)
- Added systemd service files.
- Added config file option "rabbitmq-maxpriority" that translates to the "x-max-priority" property that is added to outbox queues
- In the AMQP header the "priority" property is now read, and written back to messages that are published back to the outbox
- Added config file option "rabbitmq-declare" that can be set to false to disable declaring exchanges and queues on startup
- We now support a throttle to set the max number of bytes that may be sent per minute
- Support for the "--list-cluster" command line option to list all servers in the cluster (and the cluster-timeout option to set the time to wait for responses)
- Added config file option "render-useragent" that can be set to limit sending rendered emails only to clients with a specific user agent set
- Rewrite triggers can now have an optional start and until timestamp
- Added support for the operator =~ (regex match) in personalization
- It is now possible to use grouping operator in "if" conditions when personalizing
Bug fixes:
- Personalization: variable modifier were not working if they were used in string comparisons (for example: {if ($name|lower == "emiel")}. This has been fixed
- Fixed crash when a trial license was used to send mail without a subject
- Fixed issue when an email was loaded in an event loop that never becomes idle
- When the "rabbitmq-address" or "cluster-address" has multiple addresses assigned, MailerQ made too often connections to find out which one was reachable"
- Fixed memory leak in home-call procedure
- Fixed unfair distribution of jobs over the SMTP threads (almost all jobs were sent to the same thread)
- Fixed unneeded delay that sometimes occured because not all threads are using the same timestamp
- If the "download-cache" config file setting contains an invalid or unreachable address, mailerq now complains about that on startup
- No "online" events were sent when an mta-ip was again used for sending out mail, causing mta-ips to be reported as offline even when they were not
UI changes:
- The connections down list has a revive button to revive the connection.
- The connections down list shows the error that caused the offline.
- The connections down list shows the domains affected by offline.
- Added bandwidth counters.
- Improved counters look and behaviour.
- Added missing counters.
- Added domain check to capacity rule form.
Email preview module changes:
- Added "GET /useragent" www call to allow the UI to display the current and required user agent string to render emails
- RewritesList and ActionsList use a more sensible way of setting precedence.
- The email preview module shows 3 emails at once.
- from, to, tag filter was added to the email preview module.
- Fixed email address rendering for the EmailPreview.
- Email render widget can be configured with interval.
- Improved the email transition on the email preview module.
- Added "setip" rewrite action that combines the "addip" and "removeip" rewrite actions
- Email throttling settings can now be linked to more than one domain
- The command line option "--upgrade-database" has been renamed to "--repair-database" because it can not only be used to upgrade, but also for first setup of a new MailerQ environment, or to repair databases that are broken
- Missing tables are now automatically created on startup (no explicit --repair-database flag is needed for table-creation)
- The config file option "database-validate" now has a broader meaning: it now also covers whether the database should be checked it in the first place (it used to only cover whether timeouts with value 0 (meaning unlimited) were changed to timeouts with value -1 (meaning fallback)
- If a user adds the option "database-validate: no" to the config file, MailerQ will start without checking if the database is ok, this allows users to work around possible bugs in the check-algorithm, and it allows users to add columns to the database without having MailerQ complain about this
- Fixed bug in database upgrade
- Fixed bug in licenses with no ip limit
New features:
- A new state has been introduced ("process") that is used to report errors that were encountered when the email was internally processed (these kind of errors used to be assigned to the "bind" state, which was technically incorrect).
- Added "features-timeout" capacity property that controls how long a certificates (and possibly other features) are cached on a mta-ip level
- New rewrites implementation.
- Converted delegates to rewrites
- Implemented new Pattern match backend for the future flood responses & other actions based on server results
- The backend now supports both local pauses and cluster-wide pauses
- It is now also possible to change the smarthost settings using the rewrite module (not yet implemented in UI)
- Changed the parameters for setting and modifying rewrite actions, and the format in which it is stored in the database (watch out: not backwards compatible)
- New capacity setting: retry_bind_errors()
- Intro-errors now supports four values: "fallback to default", "proceed to secondary mx", "failure" and "respect error code"
Perpetual license:
- Added support of the new kind of licenses and the --fetch-license flag
- If a license is replaced on-the-fly, the updated trial setting is now also shared with the worker threads
Changes:
- Security errors (like certificate mismatch) no longer result in a connection to a fallback server
- All data from the database is now treated as UTF-8 data
- MailerQ now uses libfastjson for JSON storage instead of json-c (this means that plugins have to be adjusted too!)
- The capacity now also has a "minmessages" setting, meaning that no new connections are created if there still is a connection that has not reached its minmessages capacity
- Security errors were reported as error 7.7.4, this has been changed to 5.5.4
- The DKIM algorithm (sha1 or sha256) can now also be stored in the database, which takes away the need to do a DNS query to find out if a key is valid (property "enforce" must also be set to 1 to skip DNS lookups)
- The "matcheshost" option is no longer available in the reported SSL certificates
- On early reload, the timout now uses the ttl instead of default 10 minutes.
- The error message "missing body data" has been replaced by errors "missing body data in json" and "missing body data in storage"
- The sqlite journal file is now also kept on disk, and ownership is changed to the mailerq user (this allows one to use sqlite in a read-only directory as long as the database and journal file are writable)
- Changed enhanced error code for the message-content-could-not-be-fetched has been changed to 5.6.6
- If a storage fetch operation times out, it will no longer be treated as a fatal delivery error, but the mail will be rescheduled for later
- If a mail is parked in a temporary queue for a short while, it no longer starts a whole new attempt, but proceeds with the earlier attempt (so it does not connect to already tried IP's)
- Intro 4** errors used to proceed to secondary MX as default behavior, now we treat this as greylisting and wait a couple of minutes for a retry
- When started as CLI program, no extra thread is started to handle websockets (because in cli mode there never are websockets)
- When entire object is put on pause, all in-memory messages are now published back to RabbitMQ
- The websockets now also receive DNS events that can be used to keep an eye on the queue length for DNS lookups
- The jobs are no longer round-robin distributed over the SMTP threads, but are now given to the least active thread
- Update database/purge to clean new tables.
- When a pipelining error occured, but the DATA command was still accepted, we now close the connection to prevent that invalid mail empty mail is being delivered
- When a message is NOT accepted, we now still try to reuse the same connection for subsequent mails (we used to close the connection)
- If a "MAIL FROM" command fails, we now no longer RSET the connection because the connection was still in its initial state anyway
- The information shared on the cluster, and the data read from the cluster is now compatible with old MailerQ instances to allow running 4.* and 5.* versions at the same time
- Added "expired" counter to the counters (this number was previously included in the "nottried" counter)
- Split up the "queue" counter in a "queue" counter and an "assigned" counter to distinguish mails that are already assigned to an smtp connection, and mails that still have to be assigned
- Improved the error messages that are displayed on stdout when the application is started with a wrong or missing license file
- When a tag is paused, we now also release the in-memory mails that are associated with this tag and put them back in RabbitMQ
- Added capacity flags "greylisted_src_ip" and "greylisted_dst_ip" that control whether a greylisted mail has be be resent using the same from/to IP addresses
- The TTL setting that is fetched from DNS is now respected (Before, DNS was only refreshed once an hour if a domain was continuously kept in memory because of ongoing deliveries)
- Removed error state "amqp" (errors in the amqp state were never reported)
- The rewrite rules are now also applied to domain aliases (thus: a rewrite rule for gmail.com also applies to googlemail.com)
Performance improvements:
- Temporary queues will no longer be that often flushed between other temporary queues
- Temporary queues for paused domains are no longer started with active consumer
- Don't redeclare a tempqueue when a redeclare operation is still in progress
- Message that are loaded too soon are not decompressed or parsed as JSON, but directly published back to RabbitMQ (saves _many_ CPU cycles)
- When in-memory queues are too big, now we first publish back the queued mails for the most popular mta's (to make room for messages that can be sent out from mtas without queues)
- Improved rescheduling of almost expired messages: if the time of the next attempt would be behind the max-deliver-time, the next attempt is now scheduled for that max-deliver-time (instead of much later), so that it is removed from the queues exactly on the expiration time
- Expires messages are no longer validated / loaded from nosql (this prevents "missing body data" errors if a message is consumed after the data was already thrown out of nosql)
- The counters that are sent to websockets now run on a lower priority, so they will not block the main thread
- Multiple subsequent calls to flush internal queues are now combined into a single call, to lower the load on the main thread
- The in-memory queues are now not only checked on events, but are also analyzed every second (this fixes a possible long idle time if no event (like delivery, new connection or new message comes in) happens)
- Fixed issue with log monitor with a tag-filter (no mails were being displayed)
- The timeout setting for the "MAIL FROM" command was not respected
- Fixed possible issue with pipelining: if the connection was lost before the responses to "rcpt to" or "data" were received, it could lead to "strange" errors in the charts, and mails that were double retried (although this was prevented by the final safety check in the email::message class)
- Faster queue checking algorithm (well, hopefully), because we no longer check mails same IP's and attempts when we analyze all the in-memory mails
Added/removed config file options:
- Added config file option "database-threads" to set the number of threads that are used while the database is opened (increase this if the DKIM keys take a long time to resolve)
- The capacity that is used for temporary queues is now configurable
- Added config file option 'www-url' that can be used to manually set the URL that is exposed to the cluster (if you want to override the url that is constructed based on all the other www-* settings)
- Changed config file option "database" into "database-address" ("database" still works to be backwards compatible)
- Added options www-log-* (similar to all the other log options) to enable logging all incoming www requests
- Added config file options "www-cache-control" to add extra header to the html/css/js resources (tip: set to "no-cache, no-store, must-revalidate" on dev servers)
- Added config file options "storage-timeout" and "storage-reschedule" to control the timeout and delay time that is used in case a storage-fetch operation does not respond in time
- Added config file option "database-validate: true|false" that can be set to validate values in the database on startup (useful to remove timeouts that are set to zero which you probably never want)
- Config file variables "cache-size" and "cache-dimensions" have been renamed to "download-memory" and "download-dimensions" (old variables are still recognized for backwards compatibility)
- Added config file variables "download-blacklist" and "download-whitelist" to limit the IP addresses from which resources can be downloaded (fixes possible security issues because MailerQ could download from the internal network)
- The server-id config file variable is now automatically calculated based on the IP and port numbers of the web interface if it is not set in config file
- Added config file variable "download-ttl" to limit the time a resource is cached
- Added config file variable "download-cache" that can be set to a storage address (like mongodb://... or directory://...) and where downloaded files will be cached
- Support for config file variable "dns-getaddrinfo" that controls whether it is ok to rely on getaddrinfo() library calls for DNS lookups
Bug fixes:
- Fixed issue with locking and unlocking data that was viewed via the mgmt console (these objects stayed forever in memory, even after disconnecting the monitor)
- The sqlite database was opened using the wrong group ID (same ID as user-ID, which could be a whole different group!)
- Fixed temporary queue flushing
- Fixed default durable queue when local queue was not set
- Fixed possible webspy segmentation fault
- Fixed compatibility with async ARC calls
- Fixed crash when personalizing a message that was loaded from MongoDB
- Fixed bug in loading capacities (this only caused a crash for postgresql, but wat wrong for all engines)
- Fixed issue with storing data in NoSQL: mime data was kept in JSON instead of being written to NoSQL
- Fixed memory leaks in MTARow js component.
- Process errors are fatal. Fixes endless republishing of expired emails.
- Pause buttons from LogPreview and SMTPPreview does not stop the whole application.
- Fixed issue when multiple rabbitmq comma-delimited addresses were set in config file: MailerQ tried to connect to all of them, even when it does not even need a RabbitMQ connection (like when it was started with a command-line option as "--notify-cluster")
- Fixed deadlock when MailerQ was started with the "--notify-cluster" switch if not cluster exchange was configured
- Incoming local mail that did not have a custom routing key was not correctly accepted
- Mail that was already partially processed is no longer sent to temporary queues (this prevents that mail runs through it attempts too fast)
- Fixed the websocket event stream if a tag parameter is supplied (this parameter was ignored)
- Domains and domain-groups are no longer removed from memory if they still have messages in a temporary queue
- Fixed issue when an uncompressed message without an AMQP header was consumed right after a compressed message (the encoding was not reset)
- Fixed possible race/deadlock when destructing a temporary queue
- Fixed sending mail with a strange from-address (with a hostname that can not be turned into an organizational domain, like "pawel@meanmachine-e5570latitude")
- When an intro-error occured, this was both counted as a "not-tried" error, and as a real error
- Messages consumed from a temporary queue were sometimes endlessly pumped around
- A blocking error did not always key the MTA blocked for the configured amount of time
- Fix for potential double free in the rewrites
- Broken UTF8 data that is sent to the log websockets (the live monitor log and logfile download) is now first fixed and no longer causes errors in browsers
- Fixed minor memory leak on destruction
- Fixed crash in parsing the intercepted answer of a plugin
- If plugins explicitly rejected a message (by sending a 4** or 5** error), the mail was under specific circumstances still accepted
- The rewrite-rules were not applied to messages that were parked to a temporary queue. This has been fixed.
- In the unlikely case that a mail from a temp queue cannot be processed (happens when rewrite rules are changed in the meantime), it will now be published back to the main outbox
- Re-enable errorlog
- Fixed calculating the server-ID (this fixes local pauses if the "server-id" was not explicitly set in the config file)
- Fixed unpausing
- Fixed possible race condition when listen sockets in other threads are closed
- If application was closed the SMTP sockets were not always correctly closed
- Sometimes two HTTP responses were sent when a POST or DELETE request came in, causing the application to crash (this fixes the crash when the pause button was clicked)
- The in-memory queue of mails that waiting for a DNS query to complete, was not reported in the counter events
- Fixed bug in assigning jobs to smtp threads: it was assigning jobs to the busies thread instead of to the least active one
- Fixed issue that may have caused the warning that errors were appended to mails that were already processed
- Fixed issue that when then entire application was put on pause, the active smtp connections were not closed down
- If deliveries are paused, MailerQ did not always immediately publish back the mails that were already assigned to an SMTP connection back to RabbitMQ
- During the short period of time that a DNS lookup lasts, it was possible that the in-memory queue for that domain was longer than the max set in the capacity settings for that domain
- Fixed bug: if a message was parked to a temporary queue during a DNS lookup for a domain, it was also rejected to RabbitMQ, which lead to message duplication and double deliveries
- Fixed crash when no default capacity was installed
- Fixed bug: 4** errors during the intro state were threated as fatal error, even when the domain was configured to respect the response code
- The error codes for forced errors are now respected (all forced errors used to be treated as fatal errors)
- Fixed use of spaces in the names of temporary queues
- Fixed crash if a remove-ip action was set with a NULL IP address value
- Fixed the caching of image-dimensions when a json encoded responsive email is turned into a mime string and images have to be downloaded. This cache reduces the number of actual downloads but was not working.
- Fixed invalid negative value of the "completed" counter in the DNS widget on the management console
- Fixed possible crash when application is forced-stopped when there are still connections open
- Fixed small bug in setting the intro-error capacity flag
- Fixed queue events that were sent to websockets: duplicate events were sent when the websocket was reopened
UI:
- The API of the mgmt console has been changed, and now uses websockets to send events from the server to the browser (this removes the need to poll every second)
- Counter for the amount of times forced errors were triggered
- Implemented UI interface for new rewrites.
- If a query from the management console results in a new inserted ID, return that to the user.
- Dataprocessor worker now also supports tags data, and the dashboard supports tags preview table similar to the domains preview table.
- Optimized the web interface: MTA data is gathered in the dataprocessor worker.
- Updated functionality MTAs Preview and Row in the web interface.
- Added special nodata overlay message for the domain preview, and updated some table column names.
- Preview boxes that require total data now listen to the totals event coming from the dataprocessor and totalsseries object.
- The dataprocessor will only send events out after the first 60 data points have been received.
- LogPreview and SMTPPreview creates a WebSocket only when it's needed.
- We should count only counters events when we wait for first 60 counters events.
- Added state and attempt to LogLine.
- LogLine is color coded for error and accepted.
- Added header to logs.
- Fixed cluster status.
- redone log and SMTP monitor styles.
- Created Feed component that holds limited amount of components inside.
- Refactored log.js, monitor.js, LogPreview, SMTPPreveiw.
- It's possible to restore dashboard to factory settings.
- It's possible to share dashboard config via a link.
- Fixed errors and success numbers (no logner negative success is possible).
- It's possible to clean up after a PauseButton.
- Fixed removal of LogPreview.
- Added REST calls "/domains" and "/tags" that return all known domains and all known tags
- Fixed classes inside SMTP monitor.
- Implemented TagsPicker and used it in all places where we can type in a tag.
- Capacity timeouts don't allow to setting timeout to 0 seconds.
- Capacity timeouts will tell user that he has invalid values.
- When creating or editing flood patterns UI enforces positive integers for paused duration and reduced duration (and additional 0 for reduced).
- Inside flood pattern form the paused and reduced duration have no fallback option.
- Added handling for intro_error capacity/flood pattern setting.
- Removed handling for intro_error_means_fallback capacity/flood pattern setting.
- Added fetching data overlay for domains na MTA tables.
- Added websocket /queues that can be used to track the number of messages that have been consumed per queue, and to track the number of publish operations that are scheduled
- QueuePreview include info about outbox queue and temp queues.
- It's possible to see relevant AMQP queues (and their stats) in the dashboards.
- Flood patter forms don't contain queue configs cause they are irrelevant for them.
- Fixed mta/domain handling on the tag page.
- Fixed flood patterns 0 + default value rewrite action bugs.
- Added cluster pause functionality to the global pause button.
- Fixed handling 'connect' message in SMTPFeed.
- Added a new dashboard widget: DNSPreview.
- Fixed handling /queue socket output.
- The ReducedWarning shows also number of paused MTAs.
- Fixed odd overflow in the read only capacity.
- Added expired counter to tables.
- Added expired counter to MessagesPreview.
- Series parsing respects expired counter.
- TagsPreview tags links respect current target and localip.
- QueuePreview shows outbox and temp queus only when looking at root status.
- Textual changes to reduced warning.
- The events about consumers and publishers no longer require a special websocket, they are now sent over the regular websocket monitor connection
- Support for HTTP POST /revive call that can be used to manually disable a reduced capacity
- The CapacityPreview component will show a 'Revive' button. This button will end the current phase of a flood pattern that is installed on a domain.
- The SMTP filter reads configuraton from the URL. And updates url when the filter changes.
- The "/revive" HTTP POST call can now also be used with a localip and remoteip parameter to "unmark" an IP as offline
- The "offline" events now also receive an extra "error" property with the error that caused the ip to go offline
- Fixed the 'add IP' and 'remove IP' rewrite actions to support the new interface.
- Added support for the 'smarthost' rewrite action.
- Added additional counters: forced, noips, notsecure, nobody.
- The GET /errors and GET /error calls now have a "cluster" property (true or false) that sets whether the error applies to the entire cluster or only to this instance
- The POST /error call can have a cluster parameter that can be set to true and false to make force errors that apply only to this instance
- Added support for connection_featurestime for when editing and reviewing capacities.
- Fixed parked counter being omitted.
- Better default counters for message statistics.
- Fixed missing remove of DNSPreview.
- Added missing colors to the charts.
- Fixed building websocket url to logfiles.
- Added greylisting section to capcity/floow pattern forms.
- Added new errors chart dashboard box.
- Added additional counter: failures.
- Fixed Series implementation to properly calculate error and success rate and not clear tags and domains list in the process.
- Added support for cluster property for forced errors.
- Counter can handle NaN value.
- RemoteIPRow can hide/show the weight column.
- Fixed computing success and error rates
- Fixed how values from counters are handled in the Series, Charts, Tables, etc...
- Table headers will show the name of the column in additional tooltip (the title one).
- Implemented proper ConsumersPreview.
- Forced errors list and paused lists show the scope of the item.
- Configurable timeouts on every step of SMTP connection, which can be set in the email throttling and flood patterns
- Configurable behaviour on some of the received errors (i.e. 500 error on connect when IP is blacklisted)
- Implemented "warming limits" to ease up warming up new IPs
- Added configurable queues for local email addresses
- Updated web interface, lots of style fixes
- New default value for max number of http connections is now 100 (the old default was 10)
- Improved websocket handling in the SMTP monitor
- Fixed connection to multiple rabbitmq hosts
- Fixed regression: deadblock reoccurred when publishing and consuming from a temporary queue at the same time
- Fixed issue with handing over messages between cluster nodes
- In the SMTP monitor, hitting the enter button will no longer switch between the incoming and outgoing monitor
- Fixes crashes when accessing the management console
- Better error reporting messages for issues with license, certificates and smtp-default-ips
- The web interface to add local address was way too strict: it checked if the added address was valid, but this is not necessary for patterns
- Fixed crash when an SMTP connection came in while MailerQ was still being initialized
- Fixed crash when the garbage collector was cleaning up the list of blocked incoming IP
- Fixed accessing uninitialized variable when using manually configured MTA names from the database
- Refactored passing messages from the consumer thread to the main thread, solving race conditions between threads
- Fixed possible crash when publishing to a message to a message-specific DSN queue
- Fixed possible crash when data could not be loaded from storage
- Fixed retrieving directory listings on XFS filesystems
- Added a nottried counter to the status root, so messages with an invalid "ips" property show up in the management console
- Added an ARC check for incoming mail, configurable with the smtp-check option
- Made all the incoming SMTP threads concurrently listen on the same port, so the incoming SMTP traffic doesn't take resources from the main thread
- Added config file "smtp-mappable-ips" and "smtp-unmappable-ips" that contain ranges of IPs that can directly be mapped from incoming IP to the outgoing IP, unmappable IPs will be sent based on the smtp-default-ips variable
- The variables smtp-defaultip and smtp-defaultips are deprecated in favour of smtp-default-ips
- Small optimization in skipping personalization variables initialization
Comment out options that prevent MailerQ from starting with just rabbitmq and license (secure ports and spamassassin).
- Fixed an issue where deliveries were counted more than once in the "All IPs" overview.
- Added support for NoSQL storage compression
- Added support for DKIM failure reporting (RFC 6651)
- Expanded the use of the "smtp-defaultips" config option. If a message is injected without specifying an IP range it is now only tried on this list of default IPs.
- Improved the "dir://" storage strategy. In the new release we save the data in a series of subdirectories to make lookups more efficient.
- Fixed the smarthost SMTP header (set to "x-mq-smarthost-hostname" to correspond to the JSON property)
- Fixed an issue where the smarthost domain was linked to its MX record instead of its A/AAAA record
- Fixed a bug that caused MailerQ to randomly crash if no smtp-certificate or smtp-privatekey were provided.
- Fixed a bug that allowed a message to use all available MTA IP addresses when it had an empty array for the "ips" property.
- Updated web interface menu
- SMTP server now auto-reloads its SSL certificate and key if the files defined by smtp-certificate and smtp-key are updated
- Webserver now auto-reloads its SSL certificate if the files defined by www-certificate and www-privatekey are updated
- The license file now auto-reloads, so MailerQ doesn't have to be restarted when updating the licenseFixed an issue in the logfilter where tags with spaces in them couldn't be searched
- Added a sane default capacity setting when the user hasn't defined one
- Fixed the capacity-editing page for default capacity
- Fixed an issue where the message expire time could be set after the time the mime was removed from storage (maxdelivertime is now set in the Json when the message is first received by MailerQ)
- Fixed text overflow in error tables in the management console
- Fixed flood patterns not triggering when receiving a bounce on initial connect
- Fix for log filter tags containing spaces
- Fix for DMARC .xml.gz recognition
- Added JSON options that can be used to add/prepend/append/update/replace headers in the mime
- Added a --notify-cluster option that triggers a database update in all cluster nodes
- Made the --purge-database option behave as a CLI command, it runs a database purge and then quits
- Added a couple of exception handlers to prevent that the application crashes when the management console handlers are unable to connect to the database
- Fixed issue that caused mta-ips to go offline when an individual error occurs (like a timeout or lost connection during a DATA call)
- Fix an issue where the temporary queue would not decode gzipped messages
- Fix fatal DNS lookups resulted in a recoverable error (Note: if a host request timeouts this still results in a retry)
- Fix attempt number that was always set to 1 for retried DNS errors
- Fix greyed out pause button caused by wrong pause inheriting
- Fix prepending to group level queue: uses the limits set for the in-memory queue
- Fix wrong implementation of compression: we now use the "content-encoding" field from the AMQP envelope (as was documented) instead of checking if the "header" field from the AMQP envelope as a "content-encoding" property
- Capacity page mpw properly informs about aliases or already existing capacities
- The "max-attempts" setting is now used for the number of times a message was loaded from rabbitmq, and not for the number of different connects that were tried
- The interval between retries is now configurable ("x-mq-retries" in mime header, "retry-interval" in config file, and "retries" in json)
- Added optional gzip compression for rabbitMQ traffic
- When a mail is DKIM signed by MailerQ, it will now always be transferred using 7bit encoding, even when connection supports 8bit
- Fixed missing rows inside forced errors page in web interface
- Fixed a DKIM signing bug when message contained non-ascii data
- Fixed error in MTA IPs page when no SPF record is set for domain name.
- Fixed segmentation faults caused by a race condition between the consumer thread and the master thread.
- Fixed bug with incorrect query when using postrgesql database
- Fixed issue that sometimes caused message to stay in an in-memory queue when a domain was marked as being offline
- Mails were sometimes published back to the outbox without any errors, resulting in endless loops of republished emails
- Fixed issue with errors that were not always appended to outgoing emails if an mta-ip was already in an error state
- The outbox names now are prefilled with zeros (outbox+0001, outbox+0002, etc) so that they can be ordered by the rabbitmq mgmt console
- Expired mails are recognized earlier, so that they do not have to be pumped around a temp queue all the time
- If a domain has a lot of "Not Tried" messages the user will be warned in the management console
- Fixed errorlog description overflow and some quirks in the DKIM
- Added support for tags, viewable in the mgmt console and in the log files
- Messages can now be "tagged" with a "tags" property in the json, and the mgmt console can be used to pause mails with a certain tag
- Added forced errors for tags and made them visible/configurable in the mgmt console
- Added support for ARC signatures, a DKIM key in the JSON may now contains a protocols property holding an array with
- Via the web interface you can now specify whether a DKIM key should be used for DKIM, ARC or both
- Mails that fail because of connection failures are now immediately retried to a different IP
- 8bit messages are converted into 7bit messages if the connection does not support 8 bit
- The "8BITMIME" parameter is now sent along to the receiving server if it supports this
- More results are logged when trying to deliver a message over connections that are offline
- All the web ports now use the same ssl context
- Improved log file interface on the management console
- Display the number of parked messages for individual paused MTAs on the domains overview
- Minor display tweaks on the domain page when individual MTAs are paused
- Performance improvements to the management console
- A 4** error response on an "EHLO" message triggers a deferred delivery, instead of a fallback to a "HELO" handshake
- Fixed endless pumping of messages from one temporary queue to the other
- Non-fatal DNS lookups will not result in an immediate error, but these mails will be retried
- Non-fatal DNS lookups (like DNS server is unreachable) are faster retried than fatal lookups (hostname does not exist)
- Fixed bug that caused mails not to be delivered when primary MX is not online
- Fixed bug that caused messages to stay in memory when a delivery failed
- Fixed DKIM signatures for keys with the "always" flag set (they were processed twice)
- More STARTTLS configuration options and security feedback
- MailerQ now supports cascading capacity settings
- Added "smtp-defaultip" config file option to override the IP address from which mails are sent (MailerQ normally sends out mail from the same IP as the IP on which it received the mail)
- International email addresses and domain names, and treats them in the right way. Addresses in JSON objects that are published to RabbitMQ are always UTF8 encoded
- SpamAssassing support - mails can now be sent to the spamassassin daemon
- Various tweaks to the domain view in the management console
- The list of domains shows more warnings/errors and is sorted by relevance
- The SMTP traffic monitor can now also attach to already-connected TCP connections
- Show the hostnames of the connected servers in the SMTP monitor
- Removed invalid "edit" and "remove" buttons from the domain view for a default capacity
- The domain view now shows a list of mtas subject to flood control
- The management console now shows which flood patters are currently in use
- When the sqlite database is used, the file is now created with the right owner (file was originally created for "root")
- Fixed flood_response database structure
- Fixed segfault when retrieving envelope address from plugins when no such address was set (as in bounces)
- We no longer start an extra fetch operation for the DSN if we already know that the message can not be found in storage
- Fixed crash when constructing a delivery status notification based on data from the json
- Failed storage retrievals are now logged to the error log
- An error message is now given when a wrong storage-policy is set in the config file (a wrong policy setting was silently discarded and treated as "none")
- Fixed the error message when passing a wrong envelope or recipient address via the CLI interface (error message mentioned that the variable came from the config file, which was not the case)
- Added "force" property to DKIM keys, to force using a DKIM key, even when it does not match the settings in DNS
- Editing DKIM keys via the management console did not respect DKIM keys without an expiration timestamp
- The data that is sent to the live log monitor on the mgmt console is now identical to the data sent to the log downloader
- The "/newkey" api call now creates DKIM keys of size 1024 (bigger keys turn out to be too difficult to handle for certain receivers)
- It is now enforced that reduced capacities will actually reduce the number of deliveries (a reduced capacity that has a higher send speed than the original capacity is capped)
- In the capacity table we can now use -1 settings to indicate fallthrough
- Properly show utf8 encoded domain names in the web interface
- Added tooltip to message counters
- Fixed errorlog logfile
- Fixed --daemon mode
- Fixed bug that caused multiple identical DKIM signatures to be added to a mail if a key matched with multiple patterns
- DKIM keys are now always used for their own domain without the need to set a pattern. This is in line with what the mgmt console already said
- Fixed DKIM signatures for messages with header with empty values (like "References:\r\n")
- When running in daemon mode (using --daemon flag) the PID is now correctly stored in the lock file
- Fixed the "smtp-extract" option: variable were not (always) removed from the MIME header if this option was set
- Fixed wrong arguments in init scripts
- Fixed default values in config file
- Cipher and Certificate in result JSON now contain a "secure" property
- The nagle algorithm on TCP sockets has been disabled
- More information in JSON result objects
- Submitting mails through spool directories
- HAProxy support
- Incoming and outgoing Delivery Status Notifications
- DMARC processing
- ...and much more...
Please check our documentation for a list of
everything that is new or has been updated
- Fixed bug with sending multiple messages over single connection to MailerQ's smtp server
- Fixed bug with setting/retrieving 'use secure' setting in email throttling page
- Fixed default location of lock file
- Fixed bug with processing dot-stuffed incoming MIME messages via the SMTP port
- Fixed crash when incoming message is received before query completes
- Fixed formatting date in log files viewer
- Fixed live log feature
- Fixed web interface overview graph
- Fixed possible race condition between threads when SMTP connections are transferred between threads
- Fixed bug when incoming message cannot be correctly transcoded to 7-bit
- We now properly check the 8BITMIME feature of the receiving smtp servers
- Fixed skipped X-mq-* headers
- Fix the DKIM key check and only start consuming once the keys are loaded
- Fix MailerQ CLI issue where the recipient is unset
- Bcc header will now be properly removed just before sending
- OpenDKIM library is no longer needed for DKIM signing
- MTA local "EHLO" name is now refreshed every 10 minutes
- Better random-picking of IP addresses when choosing between IP's to send mail from
- Create a message ID when accepting a mail over SMTP, or when reading it from the queue
- DNS settings like timeout and number of tries can now be set
- You can now add multiple DKIM keys to single domain
- Allow adding DKIM keys to the JSON to sign an email with
- Add login delay for authentication failures
- Local email addresses are now supported in web interface
- Incoming messages that should be delivered locally can be stored in a special "local" queue
- Added "force" property to enforce that mail is sent, even when if could not be encoded for remote connection
- Fixed crash when MIME data was loaded from JSON, and at the same time could not be encoded for the outgoing connection
- Support for reading TLS information from the PROXY header
- Incoming mails that are rejected can now also be published to a "rejected" queue
- Added configuration option "bounce-envelope" containing the envelope address for bounces
- Log files can now have a max age
- Log files can now automatically be compressed when they rotate
- Any responsive errors now mark emails as fatal, with additional info about it in the failure queue
- Fixed a bug where images weren't actually downloaded for responsive, they were only retrieved from the in-memory cache
- Support for "data" property to personalize an email
- Support for "inlinecss" property to convert the CSS code of an email into inline CSS
- Added new public key for license checking
- Support for secure connections to port 465 (config file option "smtp-secure-port = 465")
- Allow a port range on the smtp-port and the smtp-secure-port
- Allow a port range also for the built-in web server, and allow listening both for plaintext and tls
- Allow specifying the allowed ciphers for TLS
- Added the www-ciphers option to the configuration, to override the ciphers used for the internal web server
- When running in CLI mode, no connection to NoSQL is made, all messages are initially fully stored in RabbitMQ
- Incoming mails are now fully stored in RabbitMQ, and not first in NoSQL
- Refactored processing of incoming messages
- Added logging of connection info on incoming SMTP connection
- Fix a small bug requesting the message id from a plugin
- Change the way plugins can override message id generation, so they can access context and connection information
- Handle incoming delivery reports in MailerQ, they will be published to the "reports" queue
- When running in CLI mode, IP check is no longer performed, so that running in CLI can be done on any server
- Update API to new specification
- Throttle failed authentication attempts in MailerQ
- Added plugin support for authentication
- Added the possibility to store custom properties on the context and on connections from plugins
- Fixed crash when an incoming SMTP connection was closed before a storage operation was completed
- Fixed bug with incoming SMTP connections that sent multiple "RCPT TO" instructions
- MIME data was not correctly removed from input JSON after data was stored in NoSQL
- Fix a bug that caused nodata error when using smtp server
- Integrated the Responsive Email algorithm
- Incoming SMTP connection can be secured with TLS and login credentials
- Logfile settings in configuration file has been changed
- JSON properties "body" and "keepbody" have been renamed to "mime" and "keepmime"
- It is now possible to create a global dkim key
- Refactored setting custom HELO/EHLO hostname
- Fixed bug in setting envelope in smtp-mode
- Fixed bug in local ips in smtp-mode (it caused 'bind' error)
- Fixed bug in rpm package
- Cleaned up debug code
- Fixed bug in smtp-range
- Fixed bug in initialization and cleanup
- Smarthost support added
- Implemented plugin architecture
- Fixed a bug in regex flood patterns with negative lookahead (?!)
- Fixed regex flood patterns
- Disabled usage of SSLv3 for web interface. It now only uses TLS (1.2 / 1.1 / 1.0)
- Fixed bug with some messages being discarded when using CLI mode
- Added support for new OpenSSl versions
- Added missing button to add a limit on the email
- Fixed possible infinite loop in delegation
- MailerQ always used all ips for a receiving domain, even the fallback ips
- SMTP monitors can now switch on/off whether bodies should be sent
- Fixed an error in hash algorithm for domain groups causing hash collisions. That resulted in a lot of 'relay denied' errors.
- New feature: Delegations. It is now possible to delegate all or some messages from one mta ip to another (for example if the ip got gray/blacklisted).
- Editing capacities via the web interface has been improved.
- Mta IPs and domains web pages now uses JSON arrays (that lowers amount of data sent from server).
- When a mail could be sent from multiple MTA's, and one of them was put on pause, the mail is now first tried to be delivered from one of the other MTA's.
- The ajax call that is used by the drop down boxes for selecting an MTA and selecting a delegate have been made faster.
- The initial data that is sent to the chart has been optimized.
- Pausing a domain was not working, this has been fixed.
- Fixed bug in IP addresses/ranges when no mask was supplied.
- Fixed buffer overflow on optimized Debian code.
- Fixed autoincrement columns for Sqlite.
- Application was not closed correctly when cluster was not running.
- When a MailerQ instance closes down, it announces that to the cluster, so that other cluster nodes will no longer send messages to it.
- Max idle time was passed on in milliseconds from the config file, this should have been seconds.
- Support for HTTPS connections for the management console.
- Dropdown with cluster nodes is now alphabetically ordered.
- Optimized inter-thread communication.
- Connection state is now also logged for errors of type "lost".
- Error log file is now appropriate user permissions.
- Solved extreme slow delivery rate, when license file contained an unlimited capacity.
- When a mail can be sent from multiple ips, MTA's that are not on pause are now first tried.
- Information on parked messages added on a per-domain and per-mta basis.
- General tweaks to the responsiveness of the interface.
- Added an aliaspicker in the domain view for easier navigation between similar domains.
- A warning is displayed on the DKIM overview page if MailerQ is configured without DKIM support
- Improved the live log monitor, it is now less likely to slow down the browser.
- Query errors when loading, storing or removing capacities from the database are now logged.
- Added error checking for log file writing.
- Fixed important bug that caused many messages to end up with a 'nodata' error.
- Emails with no body and no key are now immediately reported with a nodata error, without setting up a SMTP connection.Flood pattern was active for a too short period of time.
- Capacities were not immediately loaded (it took five minutes).
- Removing a capacity linked to a specific MTA caused a crash.
- Storing capacities was not working when MailerQ uses a Sqlite database.
- Fixed potential crash in log writer because it was accessed from multiple threads.
- The initial log file was opened for user 'root', this is now chown'ed to the right user.
- Minor interface bugs fixed.
- AMQP channels now run in separate threads making them run faster.
- Cluster configuration options defaults (empty) now point to the same rabbitmq instance as outbox queue.
- New options to bind web interface and SMTP server to specified IP addresses.
- Different log behavior, now logs are rotated when they reach certains size. (100MB by default)
- Fixed race conditions causing fatal crashes.
- Fixed buffer overflow in network buffer.
- Fixed bug in PostgreSQL queries.
- Added missing files in www dir.
- Completely refactored the code.
- New underlying AMQP-CPP library.
- Implemented flood patterns.
- New user interface.
- Fixed issues with online/offline status of domains, exchanges and ip addresses.
- Mails were sent even before the capacity for a domain/local IP combination was fully loaded.
- When exchanges and IPs change while MailerQ is running, we now also update the online status for the exchanges and domains.
- Logfile was not parsed correctly, which caused remote IP addresses not to appear in management console log viewer.
- Fixed delivery results multiline message order in log.
- Different TCP connections are used to publish and consume RabbitMQ messages to speed up publishing.
- Added ajax call /ajax/topology to retrieve the entire topology structure in JSON format.
- The pause buttons have been refactored to simplify code, and it now is possible to pause much more deliveries via the management console.
- Management console upgrade to jQuery 1.10.1
- Added pause delivery feature for whole application, local IPs, receiving domain names/IP or combination of local IP and receiving domain name/IP.
- Fixed bug when domain was marked offline infinitely after network error.
- Fixed delivery limits per ip.
- Fixed race condition that might have caused MailerQ to stop consuming messages from RabbitMQ.
- Fixed delay when republishing message, now delay increases with number of retries.
- Fixed handling of ConnectionLost results, so they're not retried infinitely.
- The 'nohosts' result will no longer trigger a retry.
- Multiline answers are now merged into one answer before they are sent to the log and message queues.
- Fixed showing error on failed storage connection.
- RabbitMQ Connection error now shows more specific error message.
- Added custom x-mq- headers to be included in JSON message.
- Added support for SHA256 hashing algorithm.
- Added dkim key verification.
- Fixed possible segmentation fault when signing DKIM with multiple threads.
- Fixed bug when marking well known servers like gmail as being offline.
- Fixed variable mismatch in logentry class that caused log search to show wrong output.
- Fixed bug that did not allow to turn off results queue.
- Outbox queue is now optional. If set to empty string, MailerQ will not send any messages.
- Defaults for max-deliver-time and max-attempts are now config file options.
- Added option to turn of persistent delivery mode for RabbitMQ messages.
- Allow "@" character in database and storage passwords in config file.
- Fixed sqlite database update query.
- Added more descriptive database connection error messages.
- Fixed init scripts to close MailerQ nicely, so the closing messages are in the log.
- Fixed crash that occurred because mail was logged after it was destructed.
- The "bind" error is detected earlier in the process, because MailerQ already knows which local Ips will work.
- If we receive more than 20% of the time a strange response from a server with at least 5 msgs, we consider the server strange and mark it offline for a while.
- For every extra result added to the results array in the JSON object, we now publish a msg to the results queue
- Added support for DKIM signing.
- The "domain" property no longer has to be included in the JSON object, MailerQ now filters it out from the recipient's email address.
- The "retry" result type has been split up in the "idle", "reuse" and "reset" result types.
- Status page now holds publish counters for all queues that are being used by MailerQ.
- Log can be filtered on 'failure' to list all possible failures.
- Log now contains entire recipient's email address, not only the domain name.
- Failed deliveries that are going to be retried are marked with an asterisk (*) in the log.
- Fixed possible segmentation fault when multiple emails were received over the same connection.
- Instructions were sent too early to worker threads (when they were not yet initialized), which could result in initial instructions being missed.
- Improved validity checks for email addresses and domain names. Checks are more tolerant now.
- Added PostgreSQL and SQLite support.
- Mysql, PostgreSQL and Sqlite can now be used as Couchbase alternatives.
- All couchbase options in the config file have been replaced by a single storage option.
- All database options in the config file have been replaced by a single database option.
- A retry queue can be used to which all retry attempts are published.
- Polished mailerq web interface and merged domains view with capacities view.
- Mails are no longer published to NoSQL at the first attempt (only for subsequent attempts).
- It is possible now to set specific limits for localips in combination with domains.
- Added new charts with live activity of deliveries over last minute, and deliveries for the last month.
- Changed behaviour of the search bar in log entries list.
- Reimplemented log page in management console with new feature to search for log entries
- Now list with log entries will autoload when scrolled down.
- Live log refreshes itself every 30 seconds.
- Sockets are opened before any files are created to prevent creating files as root.
- Fixed statistics data storing on exit.
- Fixed error in rpm package init.d script.
- Fixed error messages.
- MailerQ can be started in command-line interface (CLI) mode to send individual emails.
- Mail submitted over SMTP can now also be ended with ".\n" instead of the official ".\r\n".
- Delayed mails are no longer sent to temporary queues in RabbitMQ, but to fixed ones. The number of queues has been reduced and a "delayed" timestamp is now included in the JSON message.
- Mails sent to the SMTP port can now hold x-mq- properties that are mapped to JSON properties in the message queue.
- Config file variable smtp-range can now also be a semicolon separated list of IP ranges.
- MailerQ can now deal with failed storage operations (previously it assumed that Couchbase was reliable, but sometimes Couchbase reports a failure).
- SMTP server now accepts a message for delivery first, even before it is confirmed that it has been stored in Couchbase and RabbitMQ (this speeds up receiving mail).
- Log on management console now holds the entire log.
- MailerQ administration console retrieves minified scripts.
- When a message is delayed, the connection monitor showed a full timestamp instead of a float.
- Management console no longer shows "console.html" in the address bar.
- Index.html and console.html for the management console are no longer cached (which caused login problems).
- Fixed solving email addresses between <> characters.
- The queue counters for RabbitMQ could get out of sync when mails were retried.
- Solved deadlock when running multiple TLS sessions in parallel.
- Added support for libcouchbase 2.0.5.
- Added SMTP interface.
- Added live SMTP connection monitor in the management console. Now it is possible to monitor the activity of the server through a websocket.
- Added logging of errors and stack backtrace on segmentation fault to file (by default /var/log/mailerq/error.log).
- Support for ESMTP pipelining added.
- SMTP answers that are longer than 10kB without end-of-line are now considered errors.
- Added filterbar on top of the management console page.
- The start date in status page was showed in UTC. Now it shows the time in the local time.
- When options are missing in the config file, MailerQ now uses builtin defaults instead of empty values.
- MailerQ needs to start initially as root only if using ports restricted for root only(<1024), or switching to a different user (by default: mailerq) after it was started.
- Broken RabbitMQ connection now triggers error message and application stops.
- Solved problem with finding local IP address.
- Fixed showing retries in management console charts.
- Fixed handling smtp connection establishment errors.
- Fixed issue with chart data not retrieved correctly from the db, when querying with ip filters.
- Fixed issue with ajax call timers in administration console.
- Fixed crash when retrieving an IP address of a network interface without an address.
- Fixed crash when application was closed down while still waiting for a resolver answer.
- Fixed crash when answer from resolver was received after timeout.
- Added english (en) as language to management console, so that Google Chrome will no longer recognize is as Spanish.
- Split up HTML code in management console over multiple HTML files.
- Lazy initialization of javascript code of management console and use of global variables refactored.
- Fixed issue with mutex locks that might have caused MailerQ to crash.
MailerQ 0.3.0 [2013-03-11]
- Emails that should have been retried (greylisted) are no longer treated as final failures.
- When the very first SMTP message received contains a 4* code, we treat that now as greylisting, and no longer as unexpected input.
- The JSON encoded object can now also hold properties that contain the queues to publish the results to.
- The HELO hostname can now also be set in the JSON structure.
- The result JSON objects now also hold 'secure' property if the message was sent over a secure connection.
- The result JSON objects now also hold 'from' and 'to' IP addresses.
- Added many possible result types which were not included in the result JSON objects.
- New configuration options: max-memory and max-messages.
- The timestamps in MySQL database are stored in UTC for better timezone support.
- IP addresses are now stored in binary form in MySQL.
- In management console the status shows a chart of the deliveries.
- HTTP responses now show "MailerQ Management Console" as server name.
- Messages from RabbitMQ are now only acknowledged with their DeliveryTag, which saves memory.
- Every email that holds the entire body is now immediately copied to Couchbase and republished to RabbitMQ to bring down memory consumption.
- Brought down memory consumption by not copying all properties of the JSON object.
- Libmagic resource is now shared by page views, so libmagic does not have to be initiated for every single download
- Added automatic detection for broken RabbitMQ connections.
- Prevent consumption from queue if no consume() call was done.
- MailerQ no longer depends on libmicrohttpd.
- Dynamically loading of libraries no longer only try to open "library.so", but also specific version numbers that are known to be compatible.
- Fixed potential buffer overflow when looking for newlines in received SMTP data.
- Fixed segmentation fault when sending a large buffer.
- Fixed crash when SyncWatcher was destructed from inside its handler callback method.
- Fixed potential segmentation fault if connection got destructed when handler was called.
- Fixed memory corruption when initializing SSL (which resulted in MailerQ not being killable with CTRL+C).
- Fixed TcpLinkHandler::onLine() method which was called with a length parameter that was one byte too low.
- Solved possible race condition if a message was published to multiple queues.
- Solved possible race condition in MutexLock.
- Fixed possible memory corruption when sending a large buffer.
- Result "offline" is now only used when no send attempt is started because MailerQ already knows that the server is offline.
- Added result "unexpected" when the remote server sent an answer that we did not understand.
- Added result "timeout" which happens if the remote peer does not reply within a reasonable time.
- Added result "lost" that indicates that the TCP connection was lost before the message could be sent.
- Added result "connect" that indicates a failure because no TCP connection could be set up.
- Fixed an issue when a socket could not be bound to an IP address, the application ended up in a infinite loop.
- Added separate queues for successful and failed delivery results.
- Added option to disable result queues.
- MailerQ deamon now uses user mailerq.
- New defaults in the default config file (everything is unlimited, apart from number of connections to same IP).
- In administration console: removed scrolling of the page when clicking on a filter link.
- Fixed broken table.js script. Now the pagination of tables is correct.
- Fixed errors in messages displayed by the administration console.
- The administration console is now fully working.
- Capacities were not correctly loaded because of a query error. This is fixed now.
- When the database connection is lost, the MailerQ application now automatically stops.
- Fixed configuration files parsing. Now accepting spaces after values.
- Fixed several possible memory leaks.
- Faster discovery that RabbitMQ server is offline.
- Fixed publishing to delayed queues. This sometimes failed because the queues expired before all messages were sent to it.
- Fixed closing down MySQL connection.
- Fixed sql library initialization, the library is no longer initialized on module load, but when it is first called.
- Fixed memory alignment warnings, and solved possible memory corruption issues.
