HttpCaller

The HttpCaller is a Linux application for making outgoing HTTP calls. It reads instructions from a RabbitMQ message queue, and turns them into HTTP POST calls. We wrote this application for MailerQ users who asked for a solution to send the results of the deliveries to a HTTP endpoint. It is however a generic application: it can be used for all sort of calls to HTTP endpoints.

Installation

The application is stored in our APT repository. If you have already enabled our APT repository, you can install it with this command:

sudo apt install mailerq-httpcaller

Integration with MailerQ

A typical use case is to route the MailerQ delivery results to this application. However, MailerQ output is not compatible with HttpCaller input. The output of MailerQ must thus first be converted. We have a special application for this: mailerq-amqphttp. There is also a simple installation script that runs the entire stack of the httpcaller and the converter: mailerq-webhook.

Running

You typically run this program from the command line. It reads all its input from a config file stored in /etc/copernica/httpcaller.txt. However, you can also pass command line arguments, for example:

httpcaller --rabbitmq-address amqp://user:password/hostname/vhost

Configuration

All options can be supplied in the system wide config file, via environment variables, and as command line options. The config file is stored in /etc/copernica/httpcaller.txt. The following options are all equivalent:

• the setting "rabbitmq-address" in the /etc/copernica/httpcaller.txt config file
• the command line argument "--rabbitmq-address"
• the environment variable "HTTPCALLER_RABBITMQ_ADDRESS"

All other configuration settings can be converted in a similar manner.

Supported options for RabbitMQ

HttpCaller reads all instructions from a RabbitMQ message queue, and writes the results back to RabbitMQ. The following options can be used to configure this:

• rabbitmq-address: the address in amqp://user:password@hostname/vhost format.
• rabbitmq-api: the REST API endpoint of the RabbitMQ server in http://user:password@hostname:port format.
• rabbitmq-queue: the name of the queue from which messages are read.
• rabbitmq-exchange: the exchange to which the results of the HTTP calls are published.
• rabbitmq-overflow-queues: whether to use overflow queues, can be "true" or "false".
• rabbitmq-queue-type: the type of queues to use for overflow queues ("classic" or "quorum").
• rabbitmq-routingkey-delayed: the name of the routing key for delayed messages.
• rabbitmq-routingkey-failure: the routing key for failures.
• rabbitmq-routingkey-success: the routing key for successes.

HttpCaller needs access to a RabbitMQ server via the AMQP address to consume and publish messages. But because the AMQP lacks the ability to find out which queues are active you must also specify the REST endpoint of your RabbitMQ server.

All messages are read from the configured queue, and the results (whether the HTTP POST call was a success or not) are published back to an exchange using routing keys "delayed", "failure" or "success". If you do not like the default routing keys, you can override them with other values.

If there is a big queue with outgoing POST calls for one specific domain, HttpCaller temporarily creates an overflow queue and parks all those calls in that queue. By doing so, it can still process other messages from the same queue for other domains. This prevents that lag for one domain limits the rate for other domains. To disable this feature, you can set the "overflow-queues" option to false.

Capacity settings

HttpCaller can send out many HTTP POST calls at the same time. The limits are configurable:

• max-outgoing-connections: max total number of simultaneous outgoing HTTP calls.
• max-host-connections: max total number of simultaneous outgoing HTTP calls per host.
• timeout: timeout in seconds for a HTTP call.
• offline-ttl: number of seconds a website is marked as "offline" for failed calls.
• max-retry-attempts: number of times a failed call is retried (published back with a "delayed" routing key)

The host connections limit can be overridden on a per-call bases.

Signing outgoing connections

Outgoing HTTP connections can be signed to allow receivers to verify that the call came from the expected source. There was a proposol for a standardized way to add such signatures to HTTP calls, and the httpcaller implements this protocol. If you only use httpcaller to make calls to your own servers over an internal network this feature might not be necessary, but for calls that go over the internet it can be helpful to avoid spoofing.

• private-key: path to a file holding a PEM file with the private key
• key-id: the key ID, the specs leaves in the middle how clients should interprete it, but it could for example be used to retrieve the public key

If you do not set these options, the HTTP calls will not be signed. The signing key can also be set per call (see below).

DNS options

By default, the httpcaller uses the default DNS system resolver based on the settings from /etc/resolv.conf. But if you want to use a different DNS server, the following settings can be helpful:

• dns-server: IP address of the nameserver.
• dns-buffersize: size of the UDP buffer (default not set / use system defaut for udp buffers)
• dns-timeout: timeout in seconds for a DNS request / time to wait for an answer (default 60 seconds)
• dns-interval: time in seconds between dns query attempts / time to wait before next nameserver is tried (default 2 seconds)
• dns-attempts: number of DNS attempts before giving up (default 5)
• dns-capacity: number of simultanious DNS operations (default 100)
• dns-rotate: whether to rotate calls to nameservers (default false)

In normal circumstances there is no need to change any of the DNS settings.

Blacklist/whitelist options

To prevent that httpcaller makes calls to forbidden webservers, you can install blacklists and/or whitelists.

• ip-blacklist: IP addresses to which no HTTP calls are made (default set to the internal IP ranges).
• ip-whitelist: IP addresses to which HTTP calls are allowed (default not set: meaning all IP's are allowed).

By default, httpcaller is configured with the following blacklist: "192.168.0.0/16,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12". This means that all calls to the internal network are blocked, and that the application can only be used for calls to internet sites. If you use httpcaller for posting data to an internal endpoint, this must be overridden in the config file.

Other options

To be documented

JSON message format

The messages consumed from RabbitMQ must be JSON encoded, and must have a (minimal) form similar to this:

{
  "url": "http://www.example.com",
  "data": {
    "field1": "value1",
    "field2": "value2"
  }
}

The above instruction will be turned into a HTTP POST call to http://www.example.com with the supplied data as message body (converted to a "application/x-www-form-urlencoded" encoded message body). Alternatively, the data can also be supplied as string. But then you also pass a content-type as that cannot be auto-detected:

{
  "url": "http://www.example.com",
  "data": "textual post data",
  "content-type": "text/plain"
}

If you want to pass additional headers with the POST call, you can add that to the JSON as well:

{
  "url": "http://www.example.com",
  "data": "textual post data",
  "content-type": "text/plain",
  "headers": {
    "x-my-extra-header": "my-value"
  }
}

The config file can be used to restrict the number of outgoing HTTP connections, but this can be overridden on a per-message basis:

{
  "url": "http://www.example.com",
  "data": "textual post data",
  "content-type": "text/plain",
  "connections": 10
}

If you want to sign outgoing requests (see above), you can also override the settings from the config file:

{
  "url": "http://www.example.com",
  "data": "textual post data",
  "content-type": "text/plain",
  "sign": {
    "id": "the key id",
    "key": "private key in PEM format"
  }
}

JSON result format

HttpCallers posts the results of all calls back to RabbitMQ where you can inspect them. For each result, the original message is posted with an extra "results" array, holding one object for each attempt:

{
  "url": "http://www.example.com",
  "data": "textual post data",
  "content-type": "text/plain",
  "results": [{
    "ip": "1.2.3.4",
    "port": 80,
    "code": 200,
    "time": 1692108936,
    "duration": 0.210,
    "error": "optional error message",
    "header": "full HTTP response header"
  }]
}