Introduction
This article explains how to feed your MTA’s inbound SMTP transaction data in real-time to Abusix, which helps improve the accuracy of our Abusix Mail Intelligence product. This protocol only applies to the third wizard set-up option, “Send transaction feeds” in app.abusix.com.
SMTP Transaction Data
The data we request is simple to retrieve from most MTAs and, by design, does not include any data that would cause any privacy issues.
Data for each SMTP transaction, created on every SMTP MAIL command, is what is requested. Ideally, this data should be sent immediately after receiving the MAIL command, provided it is only sent once per transaction.
The protocol is designed as fire-and-forget UDP to minimize overhead on your mail system.
Format
Each data field must be separated by a line-feed character (LF \n) without a trailing line-feed, and all data must be sent in a single UDP packet.
While all fields are required, they may also be empty if the data is unavailable. The feed receiver will discard data where (a) there are not enough line feeds, (b) the Feed ID is not recognized, or (c) the checksum does not match the checksum we compute on reception.
Since this is fire-and-forget UDP, there needs to be feedback on whether the data being sent is valid and in the correct order or any acknowledgment that the data was received.
Data | Description |
Feed ID | This will be provided to you by Abusix and is used to identify which feed the data is coming from and is used as a lookup key to find the shared key used to compute the checksum. |
Timestamp | The UNIX epoch time in seconds that the connection was received. |
Port | The TCP server port that the connection was received. e.g., 25, 587, 465. This is used to distinguish between MUA->MTA traffic and MTA->MTA traffic. |
IP Address | IPv4 or IPv6 address of the SMTP client. |
rDNS | Reverse DNS name of the SMTP client; multiple names should be separated by commas. |
HELO | HELO/EHLO sent by the SMTP client. |
ESMTP Flag | ‘Y’ if the SMTP client sent EHLO, ‘N’ if the client sent ‘HELO’. |
TLS Flag | ‘Y’ if the SMTP client used TLS, ‘N’ if the client did not use TLS. |
AUTH Flag | ‘Y’ if the SMTP client is authenticated using SMTP AUTH, ‘N’ if not. |
MAIL FROM domain | The right-hand side of the ‘@’ of the SMTP MAIL FROM command. In the case of a null-sender or an unqualified user, e.g., MAIL FROM:<postmaster> or MAIL FROM:<>, the full argument can be sent. |
Extended JSON | For future use - this field should be empty. |
Checksum | This is an MD5 hash computed by taking all of the previous data, including the line-feed separators, including a trailing line-feed, along with the shared key supplied to you by Abusix added to this.
e.g. checksum = data + “\n” + shared key
Once this is computed, the following is then sent over the UDP socket to the feed receiver:
data + “\n” + checksum |
Example
Here is an example of the data showing each field and the format. Line feeds are shown as ”\n” for illustration purposes:
txnNNN\n
1540299376001\n
25\n
127.0.0.1\n
localhost\n
this.is.a.test.helo\n
N\n
N\n
N\n
test.com\n
\n
8c86e0ab24415f726e4def79ce9502c5
Transport
After completing all the Data Channel configuration steps in app.abusix.com, you can send us the data.
First, verify connectivity by running the following command:
$ nc -v -u -z -w 3 smtp-rttf.abusix.com 12211
Connection to smtp-rttf.abusix.com 12211 port [udp/*] succeeded!Example Code
We provide example code for Exim and a fully functioning Postfix Policy daemon, written in Node.js, which can be used with any version of Postfix. These examples may also be used to write an integration with any MTA in any programming language.
Exim
Each distro package in Exim is different, so you will likely need to customize this appropriately for your installation.
This is what I did on Debian/Ubuntu Linux.
In /etc/exim4/conf.d/acl/00_exim4-config-header - I added the following lines to the top (as macros); these must appear before the "begin acl" directive:
ABUSIX_HOST=smtprttf.abusix.com
ABUSIX_PORT=12211
ABUSIX_FEED_ID=<Your Feed ID>
ABUSIX_FEED_KEY=<Your Feed Key>
Then in 30_exim4_config_check_mail we want the “udpsend” to run for every message, accepted or not:
### acl/30_exim4-config_check_mail
##############################
###
# This access control list is used for every MAIL command in an incoming
# SMTP message. The tests are run in order until the address is either
# accepted or denied.
acl_check_mail:
accept
set acl_m_txnfeedraw =
ABUSIX_FEED_ID\n${tod_epoch}\n${received_port}\n${sender_host_address}\n${sender_host_name}\n${sender_helo_name}\n${if eq{${substr{0}{1}{${received_protocol}}}}{e} {Y}{N}}\n${if eq {$tls_in_bits}{0} {N}{Y}}\n${if eq {$authenticated_id}{}{N}{Y}}\n${sender_address_domain}\n\n
set acl_m_txnfeedkey = ${acl_m_txnfeedraw}ABUSIX_FEED_KEY
set acl_m_txnfeed = ${acl_m_txnfeedraw}${md5:${acl_m_txnfeedkey}}
udpsend = ABUSIX_HOST:ABUSIX_PORT:${acl_m_txnfeed}Postfix
The following linked Postfix Policy daemon is written in Node.js, is fully functional, and can be used with any version of Postfix.
Sending Abusix your “Spamtrap” or “This Is Spam” messages
If you wish to send Abusix, your “Spamtrap” or “This Is Spam” messages, see Submitting data via email to Data Channels and Submitting data via email to Data Channels
Learn more about Data Channels
This article will help provide you with an Overview of the XARF Format
Send us a message
Having trouble with your setup or a technical issue? Get in touch with our team of Abusix experts.
Click the chat button at the bottom and send us your questions. Alternatively, you can email us at support@abusix.com
also, follow our LinkedIn Channel for updates & subscribe to our YouTube Channel for the latest Abusix how-to-videos.