Running the Comparison Tool

Running the Tool

In this mode, you run the “Comparison Tool” passing in the API_KEY for querying Abusix Mail Intelligence (you can find the key at app.abusix.com), along with the DNS list you wish to compare against, and the file containing the IP addresses to check.

Each lookup will cause the tool to output a CSV (comma-separated values) format log-line for later analysis, so you need to redirect stdout to a CSV file.

Example:

$ ./ami_compare_linux --apikey API_KEY --list bb.barracudacentral.com ips_to_test > results.csv
Processed 27907 items
Processed 28104 items
Processed 28261 items
Processed 28367 items
Processed 28505 items
                             Blocked     %     Unique     % Blocked WL     %
Abusix Mail Intelligence       21125  74.1      20234  95.8          0   0.0
bb.barracudacentral.org         1139   4.0        248  21.8          0   0.0

As you can see above, when the tool runs, it will periodically output how many lookups have been completed. Once it is complete, it will output a summary and exit. The summary table contains the following fields:

Field	Description
Blocked	The number of IPs found to be listed in the blocklist, along with the percentage.
Unique	The number of IPs where the IP was only found to be listed on one list and not the other, along with the percentage.
Blocked WL	The comparison list shows the number of listed IPs that were found to be listed in the Abusix Mail Intelligence Whitelist, along with the percentage.

The "results.csv" file can be loaded into most Spreadsheet applications, and using "Auto Filter,” you can analyze all of the results to look for false positives and to investigate the results of the lookups.

Real-time/Streaming Mode

One of the drawbacks to using the comparison tool with log files is that whilst it provides a great way to quickly and very simply make a comparison between Abusix Mail Intelligence and another blocklist, it doesn’t provide a replica of what would happen if you were to add Abusix Mail Intelligence into your SMTP server, so we added the real-time/streaming mode to provide this.

To be able to use this mode, you need to be able to extract the IP addresses hitting your MTAs in real time.

This could be as simple as using UNIX “tail” and “grep” commands for a single node or system that uses a centralized log server. The only requirement is that whatever you use can extract one IP per line.

In real-time/streaming mode, you use UNIX pipes to do the necessary extraction and then place the comparison tool at the end of the pipe and provide ‘-’ as the filename to tell the tool that it should read the input from stdin instead of a file.

For example, this is how to extract the necessary data from a Postfix log:

$ tail -f /var/log/mail.log | grep --line-buffered -P '\bconnect from\b' | grep --line-buffered -Po '\d+\.\d+\.\d+\.\d+' | ./ami_compare_linux --apikey API_KEY --list bb.barracudacentral.org - > results.csv
Processed 696 items, errors 0, cache hits 0, queue length 115


                               Count     %     Unique     % Blocked WL     %
Abusix Mail Intelligence         663  95.3        240  36.2          0   0.0
bb.barracudacentral.org          423  60.8          0   0.0          0   0.0
Not Listed                        33     -          -     -          -     -

💡

Tip: If you use GNU “grep,” add the “--line-buffered” option to make grep flush its output on each line rather than buffering it. Doing this, ensures that the DNS lookups are dispatched immediately and spreads the DNS lookup load more efficiently. In this mode, the comparison tool will run forever and output the statistics every 10 seconds until you press Ctrl+C to terminate it. Once terminated the tool will output the same statistics before exiting. Also, the “results.csv” file will only have a list of IPs found in the comparison list to limit the output file size; otherwise, the output file could be huge. It is essential to monitor the “queue length” value and ensure that this number doesn’t simply increase; this would mean the DNS server(s) that you are querying is not answering fast enough, and therefore, a backlog of queries is building up. If this happens, you can try adding “--cache” to the command-line options; this will enable a basic 60-second cache, so that duplicate IP lookups will be returned from the local cache instead of relying on the DNS server cache. This will not affect the accuracy of the results but can significantly affect the number of DNS lookups required and, therefore, the number of queries in the backlog.

Learn more about the Comparison Tool

Back to the Overview of Abusix Mail Intelligence

Still, need help?

No problem; click the chat button in the bottom right-hand corner of the screen and send us your questions. Alternatively, you can email us at support@abusix.com

also, follow our LinkedIn Channel for updates & subscribe to our YouTube Channel for the latest Abusix how-to-videos.