diskhash

Online file storage URL blocklist (diskhash.mail.abusix.zone)

Description

We developed this list to list Online file storage URLs seen in the message body of spam sent to our primary traps.

This list compliments the domain blacklist as using Online file storage services like Google Drive and Yandex Disk has become a common way for spam to avoid IP and domain blacklisting by hiding behind these services.  Additionally, these services are usually very poor at handling abuse of their services.

Because it is not possible to represent a full URL in a DNS query, the URLs are normalized first, then SHA-1 hashed and the hash value is used for lookup instead of the URL.

To normalize the short URL, remove the scheme, then take only the “hostname” (lowercased) and “pathname” and then calculate the SHA-1 hash of the result:

https://drive.google.com/file/d/0B6aqsaIzsR0CZlpxYUZSWDRyRGc/view 
→ SHA1(drive.google.com/file/d/0B6aqsaIzsR0CZlpxYUZSWDRyRGc/view) 
= f947e57d2326ca86ba9bead20696a9208a7acdd6

Rspamd

The following code can be added to /etc/rspamd/rspamd.local.lua to enable rspamd to query this zone.   Note that you should replace "<APIKEY>" with your API key or set the "check_diskurls_dns" variable appropriately for your DNS namespace if you are using rsync.

local rregexp = require "rspamd_regexp"
local rlogger = require "rspamd_logger"
local rhash = require "rspamd_cryptobox_hash"
local rutil = require "lua_util"

local re_disk_urls = rregexp.create_cached('/^(?:drive\\.google\\.com$|yadi\\.sk$|disk\\.yandex\\.)/')
local check_diskurls_dns = '.<APIKEY>.diskhash.mail.abusix.zone.'

local check_diskurls_cb = function (task)
    local function find_disk_urls (url)
        local host = url:get_host():lower();
        if (re_disk_urls:match(host)) then
        return true
        end
    end
    local diskurls = rutil.extract_specific_urls({
        task = task,
    limit = 5,
        prefix = 'diskurls',
    filter = find_disk_urls
    });

    if (not diskurls) then return false end

    local r = task:get_resolver()

    for _, url in pairs(diskurls) do
        -- Normalize
        local durl = url:get_host():lower() .. '/' .. url:get_path()
        local durl_hash = rhash.create_specific('sha1', durl):hex()
        local lookup = durl_hash .. check_diskurls_dns
        local function dns_cb(_,_,results,err)
            if (not results) then return false end
            if (tostring(results[1]) == '127.0.3.2') then
                rlogger.errx('found URL %s (%s) in Disk URL blacklist', durl, durl_hash)
                return task:insert_result('RBL_AMI_DISKURL', 1.0, durl);
            end
        end
        r:resolve_a({ task = task, name = lookup , callback = dns_cb, forced = true })
    end
end

local check_diskurls = rspamd_config:register_symbol({
    name = "RBL_AMI_DISKURL",
    score = 3.0,
    description = "Disk URL found in Abusix Disk URL blacklist",
    group = "abusix",
    type = "callback",
    callback = check_diskurls_cb
});