I recently integrated ThePiepers.net with Amazon CloudFront to serve all the media. This has improved the overall site responsiveness and gives my visitors a closer endpoint.
After I setup a simple push method for publishing changes to Amazon S3, I quickly realized that it was extremely inefficient and adds unnecessary charges to my account. Enter Subversion.
The media and entire codebase is in SVN, so it seemed logical to publish the media using SVN instead of scanning the entire tree structure for changes. So, I compiled pysvn and created a Django command to diff the SVN repository for changes and publish the differences to S3.
Of course, there are a few dependencies: subversion, Python 2.6, Django, boto, pysvn and YAML. The commands will allow you to fire off as many workers as you have bandwidth to decrease the upload time for larger changes (use the --workers flag).
I could have just as easily setup a few database models to keep track of the changes, but that would have been redundant as subversion does that already. In addition, a database tracking method is not very portable. Using SVN, I can perform this command on my iMac and on my linux server without worrying about whether the database is up-to-date. Nothing beats the svn up command!
The Django Amazon S3 SVN command source code is available on github/bryan-pieper/pamazons3.
A few caveats...
Amazon S3/CloudFront does not support dynamic compression. To get around this, the command optionally compresses css and javascript files (use the --gzip flag). You can add a Django context processor to handle the client gzip support. Even though most clients support gzip, it is advised to check first.
Template processor:
def gzip_client_support(request):
client_supports_gzip = False
client_encoding = request.META.get('HTTP_ACCEPT_ENCODING', None)
if client_encoding:
client_supports_gzip = bool(gzip_re.search(client_encoding)) \
and not settings.DEBUG
return dict(client_supports_gzip=client_supports_gzip)
Template:
<style type="text/css">@import url({{media_url}}/css/thepiepers.{% if client_supports_gzip %}.gz{% endif %}.css);</style>
You are probably wondering why I put .gz before the .css extension... Apple Safari does not handle compressed content correctly if it ends in .gz even if the mime-type header is right. The command will insert .gz before the file extension to avoid the Safari glitch.
When the subversion repository returns the differences, the command uses only the additions, updates and deletions. Your repository must be up-to-date, to avoid publishing changes that are not part of the SVN repository.
Part of my push command was inspired by the Django Extensions. If you have any questions or feedback about the Django SVN S3 command, please let me know.
0 Comments
