Securing Downloads

The two modules that Lighttpd offers require that a user must first get permissions to download, and have to do so within a specified window of time after which the permission times out. The difference is in the way of getting permission: mod_trigger_b4_dl just defines a trigger URL that a user must visit before the download is permitted, while mod_secdownload validates against a token to be created by a backend application (for example, our login for paying customers). Therefore, we can use mod_trigger_b4_dl to fight deep linking and mod_secdownload to differentiate between user groups.

First, let us start with mod_trigger_b4_dl. Let us presume that we want everyone to view (well, we cannot really control that, but at least download) a certain advertisement, for example, an image at the path /ads/342hgf.gif, before they can access any of our high-quality content within the next 10 seconds. We can get this to work with the following configuration:

server.modules += ("mod_trigger_b4_dl")
trigger-before-download.gdbm-filename = "/web/internal/ad_trigger.db"
trigger-before-download.trigger-url = "^/ads/342hgf.gif"
trigger-before-download.download-url = "^/download/"
trigger-before-download.deny-url = "/sorry.html"
trigger-before-download.trigger-timeout = 10

mod_trigger_b4_dl needs a place to store the IP addresses of the users who are downloading. We have a choice between using a memcache host or a GDBM database. In this case, we use the GDBM support, which has to be compiled (refer to Chapter 1), and then we set the gdbm-filename to a database file.

Note that GDBM is available on most systems by default; if our system lacks it, we can get it from http://www.gnu.org/software/gdbm/. If we do not have a GDBM database file at the path where trigger-before-download.gdbm-filename points to, mod_trigger_b4_dl will create one for us automatically.

If we have a memcache host and libmemcache support compiled into our Lighttpd (again, refer to Chapter 1), we can alternatively use it to store the IP addresses. In this case, replace the line:

trigger-before-download.gdbm-filename = "/web/internal/ad_trigger.db"

With:

trigger-before-download.memcache-hosts =
("memcache.ourdomain.net:2345") # a list of hosts
trigger-before-download.memcache-namespace = "ad-trigger"

Then, put the memcache hosts we want to use into the list. Both the methods work. The performance impact of one method over another is negligible, so use memcache if we already have a memcache host up and running (and possibly used for other things, too); otherwise use GDBM.

In either case, the IP addresses are stored along with a timestamp, and each time the download URL is invoked, all stored timestamps are checked. Timestamps older than the trigger-timeout are discarded from the database or memcache. If the IP address is found, and the entry has not timed out, the handling of the download URL is resumed; else a temporary redirect to the deny-url gets sent out.

The following diagram shows the process:

(1) The user connects to our server and surfs to a mod_trigger_b4_dl-enabled site. If the user (2) visits the trigger URL, (3) the hit is recorded in the database and he or she can (4) access the download. Otherwise, he or she is (5) redirected to the deny URL.

In our example, a client trying to download a file from the download directory without fetching the ads/342hgf.gif image would be redirected to http://ourdomain.com/sorry.html.

Anyway, here is a table of all the configuration options of mod_trigger_b4_dl:

Now that we can fight deep linking, we will put mod_secdownload to use by allowing access to a restricted download area to our paying customers.

The mod_secdownload module was created to solve a dilemma. Static downloads can only be secured from anonymous access by HTTP authentication, which is cumbersome and inflexible. On the other hand, if we use a web application to authorize the download, we also need to push all those bytes through our web application, keeping up two connections (one to the client and one to the backend) and processing data twice, which is a bad way of spending our system resources.

Now with the mod_secdownload module, we can have our cake and eat it, too. This is done by splitting the tasks of authentication and authorization—authentication is still done in our web application, which then computes a token that is included in the URL for the download. mod_secdownload will then check the token and let Lighttpd serve the download directly if it is valid. Since the download is served by Lighttpd, we get the speed of static downloads and the authentication of our web application.

(1) The user requests the download. (2) mod_secdownload redirects the request to the web application page, which returns an authentication link (for example, as rewrite). mod_secdownload then validates the link. If the link is valid, (3) mod_secdownload rewrites the download link to the download directory, from which Lighttpd serves the download as a static file. If the link is invalid, (4) access is denied.

The whole process is quite simple, apart from the calculation of the token, which is also made easy by the fact that the computation uses some well-known algorithms.

The mod_secdownload module documentation (which can be read online at http://trac.lighttpd.net/trac/wiki/Docs%3AModSecDownload) contains three code listings in PHP, Ruby, and Python. Here is the corresponding Lua magnet implementation using the MD5 library of the Kepler Project (refer to Chapter 12 for further information):

require("md5") # for hash function
prefix = "/download/" # change this to match our configuration
secret = "change this" # as we really should
function gen_sec_link(relpath)
if not relpath:match("^/") then relpath = "/" .. relpath end
local time = string.format("%x", os.time())
local token = md5.sumhexa(secret .. relpath .. hextime)
return string.format("%s%s/%s%s", prefix, token, time, relpath)
end

In our further code, we can just generate a link to "WeLoveLighttpd.avi" using the gen_sec_link function like this:

link = gen_sec_link("WeLoveLighttpd.avi")

Which will generate a link to some URL that should look like the following:

/download/4b75945527344cf16fa08cc62ef83f51/48cec3fe/WeLoveLighttpd.avi

On the Lighttpd side, we will configure mod_secdownload to:

This is done by the following settings:

server.modules = (..., "mod_secdownload", ...)
secdownload.secret = "change this" # we still should
secdownload.document-root = "/web/ourserver/"
secdownload.uri-prefix = "/download/"
secdownload.timeout = 120

By the way, if we do not want to make our links very complicated, we may also set a cookie in the application and match mod_secdownload with mod_rewrite to get the cookie contents into our URL. For this, we need to make sure that mod_rewrite gets called before mod_secdownload and uses a special rewrite rule:

server.modules = (..., "mod_rewrite", ..., "mod_secdownload", ...)
# set a global document root
server.document-root = "/web/ourserver"
# insert a set download cookie into the URL
$HTTP["cookie"] =~ "download/([0-9a-f]){32}/([0-9a-f]){8}" {
url.rewrite = ("/download/(.*)$" => "/download/%1/%2/$1")
}
# configure secdownload
secdownload.secret = "change this, too" # well...
secdownload.document-root = server.document-root
secdownload.uri-prefix = "/download/"
secdownload.timeout = 120

Now, the user will see only the http://ourserver.com/download/file.pdf URL, while mod_secdownload will get the full URL. Also, we can make the cookie expire after 120 seconds; so the timeout of the cookie and mod_secdownload are synchronized.

We now have everything we need to authorize downloads and streams, which we will look at in the following section.