A few months ago I wrote about using lighttpd with secdownload to serve files. At that time I already mentioned running out of disk space on my server, so I had to look for alternatives. I had to choose between renting a server with more diskspace and using something different all together.

I decided to stop using my own server and using Amazon S3 instead for a few simple reasons:

• It is a lot cheaper than renting a server which is just sitting idle 99.9% of the time.
• Unlimited storage. I can just keep uploading files to S3 without having to worry about how much space I am using. Upgrading my server is just a temporary solution because in a year or so I would be faced with the same problem all over again.
• It should be far more reliable than just a single static file server.

Note that I'm just using an external file server because the primary server has a rather slow connection. It's enough to serve web pages, but not fast enough to serve large files. I don't need fast replication between the servers. In fact, I want it to be scheduled to use the spare bandwidth at night.

Switching from my old setup to S3 turned out to be very simple. In the past, I used rsync to transfer files to the file server at night and used a simple PHP function in my theme to replace the links to local files with links to the file server.
Similar tools exist for S3. Instead of rsync, there is a tool called S3Sync that can be used to transfer files to amazon S3. It took some time getting it to work though. Initially it didn't support EU buckets (it does now), and it is less resilient. Whenever the connection got interrupted, the script would display a Broken Pipe message and get stuck in retries. But thanks to a post on the forum I was able to modify the script so that it would recover from such errors. Note that by now a new version has been released which probably fixes this problem - but I haven't upgraded yet. Currently the only drawback is that there is no support for bandwidth limiting so I'll have to use another tool for that.

On the PHP side I had some difficulties getting the signing to behave correctly. Most of the examples I found online had issues with filenames containing spaces or slashes. I eventually used the function below, I can't guarantee that it is correct but haven't seen any problems with it so far.

<?php
// grab this with "pear install --onlyreqdeps Crypt_HMAC"
require_once('Crypt/HMAC.php');
 
// Amazon S3 credentials
define('S3_ACCESS_KEY_ID', 'your-S3-access-key');
define('S3_SECRET_ACCESS_KEY', 'your-S3-secret-access-key');
 
/**
 * Generate a link to download a file from Amazon S3 using query string
 * authentication. This link is only valid for a limited amount of time.
 *
 * @param  $bucket    The name of the bucket in which the file is stored.
 * @param  $filekey   The key of the file, excluding the leading slash.
 * @param  $expires   The amount of time the link is valid (in seconds).
 * @param  $operation The type of HTTP operation. Either GET or HEAD.
 */
function mymodule_get_s3_auth_link($bucket, $filekey, $expires = 300, $operation = 'GET') {
        $expire_time = time() + $expires;
 
        $filekey = rawurlencode($filekey);
        $filekey = str_replace('%2F', '/', $filekey);
 
        $path = $bucket .'/'. $filekey;
 
        /**
         * StringToSign = HTTP-VERB + "\n" +
         *                           Content-MD5 + "\n" +
         *                           Content-Type + "\n" +
         *                           Expires + "\n" +
         *                           CanonicalizedAmzHeaders +
         *                           CanonicalizedResource;
         */
        $stringtosign =
                $operation ."\n".   // type of HTTP request (GET/HEAD)
                "\n".               // Content-MD5 is meaningless for GET
                "\n".               // Content-Type is meaningless for GET
                $expire_time ."\n". // set the expire date of this link
                "/$path";           // full path (incl bucket), starting with a /
 
        $signature = urlencode(mymodule_constructSig($stringtosign));
 
        $url = sprintf('http://%s.s3.amazonaws.com/%s?AWSAccessKeyId=%s&Expires=%u&Signature=%s', 
                       $bucket, $filekey, S3_ACCESS_KEY_ID, $expire_time, $signature);
 
        return $url;
}
 
function mymodule_hex2b64($str) {
        $raw = '';
        for ($i=0; $i &lt; strlen($str); $i+=2) {
                $raw .= chr(hexdec(substr($str, $i, 2)));
        }
        return base64_encode($raw);
}
 
function mymodule_constructSig($str) {
        $hasher =& new Crypt_HMAC(S3_SECRET_ACCESS_KEY, 'sha1');
        $signature = mymodule_hex2b64($hasher-&gt;hash($str));
        return $signature;
}
?>

Of course, as these URLs are only valid for a limited amount of time, make sure the clock on your server is accurate, otherwise the download links won't work.