Popular topics: Abuse Contact Validation black

shorthash

Short URL blocklist (shorthash.mail.abusix.zone)

Status: Production
Type: SHA-1 Hash
Cloud DNS namespace: <key>.shorthash.mail.abusix.zone.
Rsync File: lists/shorthash.zone
Return Codes: 127.0.3.1
Test Points: 127.0.02, 127.0.3.1, *.test, d2e4345eef7b21a542ed6d7c3dd191585b344461 (abusix.ai/testpoint), f4d986915d728956d139397effd00fee0e3725e4 (abusix.ai/testpoint/hash/short)
Listing Duration: Approximately 5.2 days after last seen

Description

We developed this list so that Short URLs seen in the message body of spam sent to our primary traps could be blocked.

This list compliments the domain blacklist as using Short URLs has become a common way for spam to avoid domain blacklisting by hiding behind these services as it is not possible to list some Short URL domains without causing significant false-positives.  Additionally, these shortening services are usually very poor at handling abuse of their services.

Because it is not possible to represent a full URL in a DNS query, the Short URLs are normalized first, then SHA-1 hashed and the hash value is used for lookup instead of the URL.

To normalize the short URL, remove the scheme, then take only the “hostname” (lowercased) and “pathname” and then calculate the SHA-1 hash of the result:

http://BiT.do/e3s49?foo=bar&bar=baz → SHA1(bit.do/e3s49) 
= bb395cece75455415de5f3b6f75c13352586788c

Info

As this is a completely new type of anti-spam check, it will require support for this to be added to your chosen mail platform.  See below for example code for rspamd.

Rspamd

The following code can be added to /etc/rspamd/rspamd.local.lua to enable rspamd to query this zone.   Note that you should replace "<APIKEY>" with your API key or set the "check_shorturls_dns" variable appropriately for your DNS namespace if you are using rsync.

local rregexp = require "rspamd_regexp"
local rlogger = require "rspamd_logger"
local rhash = require "rspamd_cryptobox_hash"
local rutil = require "lua_util"

local re_short_path = rregexp.create_cached('/^(?!(?:[a-z]{3,11}|[A-Z]{3,11}|[0-9]{3,11})$)[a-zA-Z0-9]{3,11}$/')
local check_shorturls_dns = '.<APIKEY>.shorthash.mail.abusix.zone.'

local check_shorturls_cb = function (task)
    local function find_short_urls (url)
        local path = url:get_path();
        if (re_short_path:match(path)) then
        return true
        end
    end
    local shorturls = rutil.extract_specific_urls({
        task = task,
    limit = 5,
        prefix = 'shorturls',
    filter = find_short_urls
    });

    if (not shorturls) then return false end

    local r = task:get_resolver()

    for _, url in pairs(shorturls) do
        -- Normalize
        local surl = url:get_host():lower() .. '/' .. url:get_path()
        local surl_hash = rhash.create_specific('sha1', surl):hex()
        local lookup = surl_hash .. check_shorturls_dns
        local function dns_cb(_,_,results,err)
            if (not results) then return false end
            if (tostring(results[1]) == '127.0.3.1') then
                rlogger.errx('found URL %s (%s) in Short URL blacklist', surl, surl_hash)
                return task:insert_result('RBL_AMI_SHORTURL', 1.0, surl);
            end
        end
        r:resolve_a({ task = task, name = lookup , callback = dns_cb, forced = true })
    end
end

local check_shorturls = rspamd_config:register_symbol({
    name = "RBL_AMI_SHORTURL",
    score = 3.0,
    description = "Short URL found in Abusix Short URL blacklist",
    group = "abusix",
    type = "callback",
    callback = check_shorturls_cb
});

Was this article helpful?

Can’t find what you’re looking for?

Our award-winning customer care team is here for you.

Contact Support