Popular topics: Spam blocklist Abuse Contact ValidationPolicy blocklistWelcome listCombined blocklistDomain blocklistPotentially compromised accountsExploit blocklistRspamdWhat is XARF?

Comparison Tool

Quickly compare Abusix Mail Intelligence with another DNSBL

Preface

Testing a DNS blocklist to see how it might perform on your production mail stream can be a difficult and very time consuming task, often taking weeks and requiring considerable resources to complete.

To make this much easier, quicker, repeatable and transparent, we have created a comparison tool that you can use to get a indication of how Abusix Mail Intelligence compares to other lists that you currently use and can be run in a few hours or less and that requires no changes to your mail system.

Overview

The tool is supplied as a Linux binary and should be run on a machine inside your network that has access to the necessary DNS namespaces to be able to query Abusix Mail Intelligence and the list that you wish to compare it against.

It can only be used to compare Abusix Mail Intelligence with one other list at a time and only supports IPv4 and IPv6 lists, it does not support domain or hash lookups.

The tool has two modes, logfile or real-time/streaming.  Each mode has it's advantages and disadvantages:

Mode Advantages Disadvantages
Logfile
  • Easier to collect the necessary data
  • Very quick results
  • In-depth analysis provided by CSV output
  • Should only be used on data that is < 2 days old because each list will have different policies for listing expiry
  • Does not reflect how quickly a list reacts to spam e.g. which listed the IP first.
Real-Time/Streaming
  • Needs real-time log data
  • Can be run for as long as you like
  • Identical to running both lists in production, so a perfect comparison
  • Can be more complicated to set-up
  • Does not provide in-depth analysis, only provides IP addresses missed by Abusix.

Installation

Download the tool and make it executable:

$ chmod +x ami_compare_linux

If run without any options - it will output its usage and available command line options:

$ ./ami_compare.linux 
Usage: ami_compare.linux --apikey <apikey> --list <list> <filename>

Options:
  --version   Show version number                                [boolean]
  --apikey    Abusix Mail Intelligence API key                   [required]
  --list          DNS suffix of the DNSBL to compare against     [required]
  --debug    Write debug output to stderr                        [boolean]
  --cache    Cache result data to reduce DNS load                [boolean]
  -h, --help  Show help                                          [boolean]

Copyright 2021, Abusix Inc.
Node v8.17.0 (x64)
Using DNS servers: 1.1.1.1

Not enough non-option arguments: got 0, need at least 1

Logfile Mode

For logfile mode, the tool requires a file containing a list of IPs to be checked.   This list should be extracted from the log files of your production system(s).  The logs should be no more than 2 days old and should ideally be as recent as possible for the best results.   These can either be a simple de-duplicated list of IP addresses, or, a list containing count (e.g. occurrences) and IP addresses, with the latter being preferred as it will provide a more accurate result.

Data Preparation

Here is an example of how to extract a list of IPs using standard UNIX tools from a server running Postfix.   This can be modified to work with most logfile formats with some minor modifications.

$ grep -Poh '\d+\.\d+\.\d+\.\d+' /var/log/mail.log | sort  | uniq -c | sort -rn > ips_to_test

This will create a file called “ips_to_test” containing “<count> <ip>” where <count> is the number of times that IP address has been seen in the logs and will be sorted by the IPs with the largest number of occurrences first.

Running the Tool

In this mode, you simply run the tool passing in the API_KEY for querying Abusix Mail Intelligence (you can find this in the User Portal) along with the DNS list that you wish to compare against and the file containing the IP addresses to check.  Each lookup will cause the tool to output a CSV (comma-separated values) format log-line for later analysis, so you need to redirect stdout to a CSV file.

Example:

$ ./ami_compare_linux --apikey API_KEY --list bb.barracudacentral.com ips_to_test > results.csv
Processed 27907 items
Processed 28104 items
Processed 28261 items
Processed 28367 items
Processed 28505 items
                             Blocked     %     Unique     % Blocked WL     %
Abusix Mail Intelligence       21125  74.1      20234  95.8          0   0.0
bb.barracudacentral.org         1139   4.0        248  21.8          0   0.0

As you can see above, when the toll is running, it will periodically output how many lookups have been done so far.  Once it is complete, it will output a summary and exit.   The summary table which contains the following fields:

Field Description
Blocked The number of IPs that were found to be listed in the blocklist along with the percentage
Unique The number of IPs where the IP was only found to be listed on one list and not the other along with the percentage
Blocked WL For the comparison list, this shows the number of listed IPs that were found to be listed in the Abusix Mail Intelligence Whitelist along with the percentage.

The "results.csv" file can be loaded into most Spreadsheet applications and using "Auto Filter", you can analyze all of the results to look for false positives and to investigate the results of the lookups.

Real-time/Streaming Mode

One of the drawbacks to using this comparison tool with log files is that whilst it provides a great way to quickly and very simply do a comparison between Abusix Mail Intelligence and another blocklist, it doesn’t provide an exact replica of what would happen if you were to add Abusix Mail Intelligence into your SMTP server, so we added the real-time/streaming mode to provide this.   

To be able to use this mode, you need to be able to extract the IP addresses hitting your MTAs in real-time. 

For a single node or system that uses a centralized log server, this could be as simple as using UNIX “tail” and “grep” commands.  The only requirement is that whatever you use can extract one IP per line.

In real-time/streaming mode you use UNIX pipes to do the necessary extraction and then place the comparison tool at the end of the pipe and provide ‘-’ as the filename to tell the tool that it should read the input from stdin instead of a file.   

For example, this is how to extract the necessary data from a Postfix log:

$ tail -f /var/log/mail.log | grep --line-buffered -P '\bconnect from\b' | grep --line-buffered -Po '\d+\.\d+\.\d+\.\d+' | ./ami_compare_linux --apikey API_KEY --list bb.barracudacentral.org - > results.csv
Processed 696 items, errors 0, cache hits 0, queue length 115


                               Count     %     Unique     % Blocked WL     %
Abusix Mail Intelligence         663  95.3        240  36.2          0   0.0
bb.barracudacentral.org          423  60.8          0   0.0          0   0.0
Not Listed                        33     -          -     -          -     -

Tip: if you use GNU “grep”, add the “--line-buffered” option to make grep flush its output on each line rather than buffering it.  This ensures that the DNS lookups are dispatched immediately and spreads the DNS lookup load more efficiently.

In this mode, the comparison tool will run forever and will output the statistics every 10 seconds, until you press Ctrl+C to terminate it.   Once terminated - it will output the same statistics before exiting.

In this mode the “results.csv” file only contains a list of IPs that were only found in the comparison list to limit the size of the output file, otherwise the output file could be huge.

It is important to monitor “queue length” value and ensure that this number doesn’t simply increase, this would mean the DNS server(s) that you are querying are not answering fast enough and therefore a backlog of queries is building up.   If this happens, then you can try adding “--cache” to the command-line options, this will enable a basic 60 second cache, so that duplicate IP lookups will be returned from the local cache instead of relying on the DNS server cache.   This will not have any effect on the accuracy of the results but can make a significant difference in the number of DNS lookups required and therefore the number of queries in the backlog.

Support

If you require any support during your trial, please contact [email protected] or use the live chat on any of our websites.

FAQ

Q: Can I use the tool to compare domain or URIBL lists?    

No, the tool only supports IP lookups.  Domain blocklists work very differently to each other, some list hostnames and others have very specific domain stripping rules, so comparing lists becomes very difficult to do accurately.

Q: Can I use this tool against public mirrors of popular DNSBLs?

Yes, but you should NOT do this.  Most of the time, public mirrors are offered for free service tiers.   The tools will generate a lot of queries in quick succession, so this will usually be against their service terms and is generally unfriendly.   You should only run this tool where you have a local rsync copy of the zone, or, you pay for the service and have an API/Query key to make queries.

Q: Do you charge for this tool or the queries made to Abusix Mail Intelligence

No, the tool is designed for those running a trial of Abusix Mail Intelligence.   You should not run it if you already have a subscription as it will dramatically increase your average query count which can affect billing.

Q: Can I compare Abusix Mail Intelligence with <Insert DNSBL here>

Yes, the comparison tool works with any DNSBL that implements RFC 5782, so the DNSBL being tested against MUST return positive for a lookup to 127.0.0.2 (to prove it works).  The tool runs pre-flight checks to ensure that both lists are queryable and working and will return an error if not.

You should only run the comparison with a list that works in a similar way to Abusix Mail Intelligence e.g. it's a blocklist (e.g. a positive result should be blocked) not a reputational (where the return code denotes trust) or classification list (that classifies IPs into segments or categories).

Q: How many DNSBL queries will the tool make?

The tool will generate between 3x and 4x the amount of DNS queries as IPs input to it.

For example: if you are using log file mode and you've followed the instructions to above to create an input file containing "<occurences>, <ip>" and you have 300k entries in that file, then the tool will make 600k DNS lookups e.g. once for Abusix Mail Intelligence and once for the comparison list.  It also does rDNS lookups on each, so that will result in another 300k rDNS lookups and any positive result for the comparison list will result in a lookup into the Abusix Mail Intelligence Whitelist as well.

Q: How much DNS concurrency does the tool use?

It limits the amount of running DNS queries to 50 at time, so at most there will only ever be 50 in-flight queries.

Q: Which DNS servers does the tool use?

The tool uses whatever DNS servers are configured on the host that it is running on, so you need to ensure that the host is configured correctly and can resolve names in the namespace of the comparison list.  This is particularly important if you are running a comparison against an internal dataset that you are serving via rbldnsd (e.g. you have to have the appropriate "glue" records along with ACLs to allow the DNS servers configured on the host to allow the tool to query the list.

Q: Does the tool report any data back to Abusix?

No.

Q: Is the source code available for the tool

No - not at this time.

Q: You're supplying a binary, what exactly does the binary contain?

It's a NodeJS run-time along with all necessary modules along with the comparison tool compiled down into a single binary using the "nexe" module.

Was this article helpful?

Can’t find what you’re looking for?

Our award-winning customer care team is here for you.

Contact Support