Today we’re releasing our dynamicsites code, which we’ve been developing as a pure django solution to serve multiple sites from a single django project.
The code is here: https://bitbucket.org/uysrc/django-dynamicsites/src
Background
Django includes the Django Sites Framework as part of its contrib package, django.contrib.sites (docs). The documentation for this framework primarily describes example usages:
- Associating content with multiple sites, or a single site
- Hooking into the current site from views
- Getting the current domain for display
- Getting the current domain for constructing full URLs.
The examples presented in the documentation assume a common database, set of applications, templates, views, urls, public files, and project settings to be shared across multiple sites.
Shortcomings
There are cases where developers would like to share a single Django project installation across multiple sites, with some sites using different settings, different urls, different templates, different public files, and different data. Examples of this include domain portfolio owners, and development shops with numerous clients who sites generally share 80% functionality, but completely look ‘n feel and data.
The documentation for the Django Sites Framework does not support these cases. Furthermore, the documentation really only describes using a small number of sites (a couple), where we would like to explore a larger number of sites (hundreds, if not thousands)
Our Requirements
- Host multiple, potentially hundreds, of sites from a single Django project efficiently
- A site may have its own settings,py, urls.py, apps, templates, views, public files and data
- A single site may accept requests from multiple hostnames
- A site must be served from a single canonical hostname. Any other hostnames supported by the site must permanently redirect to the canonical hostname, preserving the request path, query args, method, protocol, port, and subdomain as best as possible.
- Multiple sites may be hosted under a single hostname by using subdomains
- A site’s hostname may use international and vanity “gTld”‘s (for when ICANN fully commits to them, that is)
- We will need to access sites in our development and test environments using different hostnames than production
- The solution should be implemented entirely in Django
- The should must function with django-nonrel on Google App Engine
A note on redirects
A typical domain portfolio will need to account for intellectual property protection, and redirecting typos or alternate keywords. These are very common use cases, for example try navigating to http://finance.yhoo.com/ and notice you’re redirected to finance.yahoo.com. (Note, as I write this, http://news.googel.com/ only redirects you to www.google.com — where’s my news??) While a very logically oriented programmer type might define these as user errors, the business would like to try our best to get the user where we intended them to go. Additionally while these types of redirects can be handled in DNS or Apache/web server configs, we’d like to use an all-inclusive Django based solution, if feasable.
Example Sites Structure
The sites presented here are all fictitious. If somebody registers these domains and fills them with spam, malware, pr0n or any other offensive material, it’s not our fault. We didn’t do it.
We’re going to mix a lot of requirements together in one big bundle, as if a marketing manager was giving them to us. We’re not looking for a one-tool-for-everything solution here, although we would like the solution to be contained in django as much as possible.
Example Configuration:
Site 1 (default site)
- Name: Corporate Umbrella Site
- Site hostname: corp-umbrella-site.com
- Which hostnames redirect here: *Every request that comes to the django project for which it cannot find a site will permanently redirect to this default site.
- Subdomains this site supports: www
- Subdomain redirects: Any non-www subdomains should permanently redirect to www.
- Hostname in dev environment: cus.dev
- Hostname in test environment: cus.test
Site 2
- Name: Site About Food
- Site hostname: about-food.com
- Hostnames redirects: aboutfood.com, about-food.net
- Subdomains this site supports: www, fruit, meat, vegetables, dairy
- Subdomain redirects: meats.about-food.com should permanently redirect to meat.about-food.com, fruits to fruit, vegetable to vegetables, and diary to dairy.
- Hostname in dev environment: af.dev
- Hostname in test environment: af.test
Site 3
- Name: About Food Subdomain Site
- Site hostname: restaurants.about-food.com
- Subdomains this site supports: None (it already has one: restaurants)
- Subdomain redirects: restaurant.about-food.com should redirect to restaurants.about-food.com, as well as dining
- Hostname in dev environment: res.af.dev
- Hostname in test environment: res.af.test
Site 4
- Name: About Food Brazil Site
- Site hostname: sobre-comida.com.br
- Subdomains this site supports: www, fruta, carne, legumes, leite
- Subdomain redirects: carnes.sobre-comida.com.br should permanently redirect to carne.sobre-comida.com.br, frutas to fruta, legume to legumes, and leites to leite.
- Hostname in dev environment: sc.dev
- Hostname in test environment: sc.test
Site 5
- Name: About Games Site
- Site hostname: about.gam.es
- Hostnames redirects: about-games.com redirects to about.gam.es
- Subdomains this site supports: None (it already has one: about)
- Subdomain redirects: Any non-about subdomains should permanently redirect to about
- Hostname in dev environment: ag.dev
- Hostname in test environment: ag.test
Step 1: Set Up A Test Project & Environment
It is assumed you can set up a test project. We’re using the django-testapp from allbuttonspressed in a Google App Engine environment, modified to include the admin interface. I’m also going to run the same tests on a mysql environment, just to be sure. If you can’t get that far on your own, at least with another datastore and environment, you probably shouldn’t be reading this article but rather a more basic intro-to-django article instead.
Hosts file
We’ll be running this test-setup on an OSX development machine, and using /etc/hosts file for hostname management. Here’s what the /etc/hosts file looks like for (most of) the above configuration:
127.0.0.1 corp-umbrella-site.com 127.0.0.1 www.corp-umbrella-site.com 127.0.0.1 garbage.www.corp-umbrella-site.com 127.0.0.1 garbage.corp-umbrella-site.com 127.0.0.1 cus.dev 127.0.0.1 www.cus.dev 127.0.0.1 garbage.cus.dev 127.0.0.1 cus.test 127.0.0.1 www.cus.test 127.0.0.1 garbage.cus.test 127.0.0.1 about-food.com 127.0.0.1 www.about-food.com 127.0.0.1 fruit.about-food.com 127.0.0.1 meat.about-food.com 127.0.0.1 vegetables.about-food.com 127.0.0.1 dairy.about-food.com 127.0.0.1 meats.about-food.com 127.0.0.1 fruits.about-food.com 127.0.0.1 vegetables.about-food.com 127.0.0.1 diary.about-food.com 127.0.0.1 garbage.about-food.com 127.0.0.1 aboutfood.com 127.0.0.1 www.aboutfood.com 127.0.0.1 fruit.aboutfood.com 127.0.0.1 meat.aboutfood.com 127.0.0.1 vegetables.aboutfood.com 127.0.0.1 dairy.aboutfood.com 127.0.0.1 meats.aboutfood.com 127.0.0.1 fruits.aboutfood.com 127.0.0.1 vegetables.aboutfood.com 127.0.0.1 diary.aboutfood.com 127.0.0.1 garbage.aboutfood.com 127.0.0.1 about-food.net 127.0.0.1 www.about-food.net 127.0.0.1 fruit.about-food.net 127.0.0.1 meat.about-food.net 127.0.0.1 vegetables.about-food.net 127.0.0.1 dairy.about-food.net 127.0.0.1 meats.about-food.net 127.0.0.1 fruits.about-food.net 127.0.0.1 vegetables.about-food.net 127.0.0.1 diary.about-food.net 127.0.0.1 garbage.about-food.net 127.0.0.1 af.dev 127.0.0.1 www.af.dev 127.0.0.1 fruit.af.dev 127.0.0.1 meat.af.dev 127.0.0.1 vegetables.af.dev 127.0.0.1 dairy.af.dev 127.0.0.1 meats.af.dev 127.0.0.1 fruits.af.dev 127.0.0.1 vegetables.af.dev 127.0.0.1 diary.af.dev 127.0.0.1 garbage.af.dev 127.0.0.1 restaurants.about-food.com 127.0.0.1 restaurant.about-food.com 127.0.0.1 dining.about-food.com 127.0.0.1 garbage.dining.about-food.com 127.0.0.1 garbage.restaurant.about-food.com 127.0.0.1 garbage.restaurants.about-food.com 127.0.0.1 res.af.dev 127.0.0.1 restaurants.af.dev 127.0.0.1 restaurant.af.dev 127.0.0.1 dining.af.dev 127.0.0.1 garbage.dining.af.dev 127.0.0.1 garbage.restaurant.af.dev 127.0.0.1 garbage.restaurants.af.dev 127.0.0.1 sobre-comida.com.br 127.0.0.1 www.sobre-comida.com.br 127.0.0.1 fruta.sobre-comida.com.br 127.0.0.1 carne.sobre-comida.com.br 127.0.0.1 legumes.sobre-comida.com.br 127.0.0.1 leite.sobre-comida.com.br 127.0.0.1 carnes.sobre-comida.com.br 127.0.0.1 frutas.sobre-comida.com.br 127.0.0.1 legume.sobre-comida.com.br 127.0.0.1 leites.sobre-comida.com.br 127.0.0.1 garbage.sobre-comida.com.br 127.0.0.1 garbage.www.sobre-comida.com.br 127.0.0.1 garbage.fruta.sobre-comida.com.br 127.0.0.1 sc.dev 127.0.0.1 www.sc.dev 127.0.0.1 fruta.sc.dev 127.0.0.1 carne.sc.dev 127.0.0.1 legumes.sc.dev 127.0.0.1 leite.sc.dev 127.0.0.1 carnes.sc.dev 127.0.0.1 frutas.sc.dev 127.0.0.1 legume.sc.dev 127.0.0.1 leites.sc.dev 127.0.0.1 garbage.sc.dev 127.0.0.1 garbage.www.sc.dev 127.0.0.1 garbage.fruta.sc.dev 127.0.0.1 about.gam.es 127.0.0.1 garbage.gam.es 127.0.0.1 garbage.about.gam.es 127.0.0.1 about.ag.dev 127.0.0.1 garbage.ag.dev 127.0.0.1 garbage.about.ag.dev
Step 2: Try To Use What Is Given To Us
Sites Framework
While it appears the Django Sites Framework does not support our needs, let us begin by seeing how far we can get with what is provided to us via django.contrib.sites. A lot of django plugins already are designed to work with the sites framework, so it would be best to integrate with it than have to rewrite/extend so many plugins.
First, I created all the sites in the admin interface:
Configuring the Django Sites Framework
Out of the box, the Django Sites Framework wants a SITE_ID defined in the global settings.py. This won’t work for us, as we want the SITE_ID to be dynamic.
Djangotoolbox.sites.dynamicsite
The folks at allbuttonspressed has a djangotoolbox project available, with a dynamic site middleware component (code). Let’s try using this with our configuration.
The first thing I notice is that the middleware is trying to find the site using the request.get_host() with the port number, unless the port number is 80 or 443. The problem I’m encountering is my development environment is running by default on port 8000, yet none of the sites I defined are defined with port 8000.
- I don’t want to define the site by port number. I’m using 8000 as my dev environment. I may use some other port later on for my dev environment and I don’t want to go back and change all my site definitions. We’ll probably run the test environment on a different port, also.
- To host multiple sites over SSL you need to have a unique IP address or port number for each site. Large hosting farms will usually host HTTPS over nonstandard ports for large numbers of virtual hosts. Again, I don’t want to have to manage these port numbers in the Site definition.
For these reasons I’m going to remove the port number all together from the domain being passed to Site.objects.get(domain=domain). Let the forking begin!
The second thing I notice is lack of subdomain support, which we’ll need. With the dynamic site module, it is stripping off “www” subdomain from request.get_host() before doing the site lookup. However, for our about-food site, which supports a number of subdomains, we need to create an individual site for each subdomain. What I need is a way to find the site, and then determine if the subdomain that came with the request is supported or not. If it’s not supported, redirect to the default subdomain for that site.
Redirect App
Before jumping in and writing our own middleware, let’s take a quick check at any possible solutions for our hostname redirecting needs.
Django provides a redirects app, django.contrib.redirects (docs). however it is not designed to work with the hostname, with specific instructions on the admin form not to use the hostname. So, out of the box, this app will not support the various hostname redirect requirements we have. This makes sense, as we should have the site serving from the correct hostname before we try to make any path adjustments.
Note the text: This should be an absolute path, excluding the domain name. Example: '/events/search/'.
Enforce Hostname Middleware
From http://code.djangoproject.com/ticket/12662 – Enforce Hostname Middleware we are led to some middleware to enforce a hostname: https://gist.github.com/283860. This will support preserving the protocol and query args, and will support allowing multiple hostnames, however in the case an unexpected hostname is found, it will redirect only to the first hostname in the ENFORCE_HOSTNAME settings. It doesn’t appear to check for port, or anything but HTTP GET requests, neither. Ultimately, it’s a good base and on-the-right-track, so let’s take what we can from it and combine it with what we’ll be taking from allbuttonspressed’s Djangotoolbox.sites.dynamicsite code.
Requirements for hostname redirect middleware
- Must maintain protocol, port, and path information when redirecting
- Must redirect to a canonical hostname
- Must redirect to a canonical subdomain, noting that a site may be configured with multiple subdomains
- If a hostname cannot be found, must redirect to the default site (preserving protocol, port, and path should the default site care to throw a 404 error, or redirect of its own)
- Should provide an admin configuration form along the lines of the django redirects app
Step 3: Let the coding begin!
I’m going to borrow parts of the dynamicsite code from allbuttonspressed to strip the port number and not create any new sites dynamically. Secondly, let’s introduce an algorithm for handling subdomains. Lastly, I’ll pull in parts from the Enforce Hostname Middleware and modify them for our test cases above. Once it’s all coded up, and tested, I’ll put the first-pass up on a github account for anyone to play with, study, and submit feedback/code patches.
Extend the Django.contrib.sites model
If you ask around for how to extend an existing model in Django, you’ll find varying opinions on implementation preference. I decided to go for one of the more discouraged routes, to dynamically extend the Site class, adding a “subdomains” field to store a comma separated list of strings, rather than creating a subclass. Why? Because I didn’t want a separate table just to store subdomains. I just really want the subdomains as a Python list to see whether the subdomain requested is supported by the site or not. I have no need for a many to many relationship and the overhead involved, both in coding time and system performance.
This is actually pretty easy to do on the model side. Just one line of code really takes care of it:
SubdomainListField(blank=True).contribute_to_class(Site,'subdomains')
Of course, the custom model field, form field, and widget, are a little more involved, basically converting the comma separated string into a Python list internally. I could have opted to simply persist a serialized form of the Python list, however I thought that would be too non-portable a solution. In the end, I’ve got a textarea added to the django sites admin form which accepts a comma separated list of subdomains, cleaning them up and validating them against Django’s URLValidator (which I realize has bugs, however the team seems to be working to fix)
I also added a “folder_name” field, as the site specific configuration and templates will go in site specific folders under a common SITES_DIR. I thought about trying to a) scan a folder at runtime and extract configuration dynamically (cool, but more work, so I skipped it) and then, secondly, trying to auto-deduce a folder name from the hostname, however in the end that left for some ugly looking folder names. So I added this extra field to the Site class so the developer/admin can define from which folder the site config will be read, along with some jQuery to auto populate the field.
An screen shot of the modified Site admin screen is presented below:
Note, for my initial implementation, I’m not looking to support dynamic subdomains, like 37signals products generally do, for example, using account names as subdomains. When I need to support {account_name}.{site_domain} I’ll come back and extend this code again to tie in with an external accounts system, likely implementing the external table I decided not to use for now. So, if you want {account_name}.{site_domain} functionality, it’s not here (yet).
The Heart and Meat Of It All: The Middleware
I put the hostname redirects in a dictionary inside of settings.py.
DEFAULT_HOST = 'www.corp-umbrella-site.com'
HOSTNAME_REDIRECTS = {
'aboutfood.com': 'www.about-food.com',
'about-food.net': 'www.about-food.net',
'meats.about-food.com': 'meat.about-food.com',
'fruits.about-food.com': 'fruit.about-food.com',
'vegetable.about-food.com': 'vegetables.about-food.com',
'diary.about-food.com': 'dairy.about-food.com',
'restaurant.about-food.com': 'restaurants.about-food.com',
'dining.about-food.com': 'restaurants.about-food.com',
'carnes.sobre-comida.com.br': 'carne.sobre-comida.com.br',
'frutas.sobre-comida.com.br': 'fruta.sobre-comida.com.br',
'legume.sobre-comida.com.br': 'legumes.sobre-comida.com.br',
'leites.sobre-comida.com.br': 'leite.sobre-comida.com.br',
'about-games.com': 'about.gam.es'
}
An admin panel for the redirects was defined as a “should” requirement, and so we’ll punt for the next revision to make one. Besides, since this redirect data will be accessed on every request, I felt it would make sense to have it defined in python code. Later we can look at caching the entire table in memcache if the settings file proves to be a headache.
Next, a function to clean up django’s request.get_host() to separate the host and the port number, and make sure the host is indeed lowercase.
def get_domain_and_port(self):
"""
Django's request.get_host() returns the requested host and possibly the
port number. Return a tuple of domain, port number.
Domain is lowercased
"""
if ':' in self.request.get_host():
domain, port = self.request.get_host().split(':')
return (domain.lower(), port)
else:
return (self.request.get_host().lower(),
self.request.META.get(SERVER_PORT))
Then, we get into the main loop, which will split apart the requested domain name, subdomain, by subdomain, until a matching site is found with the lookup function. You can check out the source code for this implementation.
The lookup function does a quick check in the cache to see if the site_id is already in the cache (ala allbuttonspressed code) and if not, checks the database for any sites matching the currently requested domain. If no site is found, the main loop proceeds to the next subdomain.
This way, we can handle international domain names and sites with numerous levels of subdomains with more capability. I couldn’t rely on any code assuming that a domain will always be structured as: {subdomain}.{hostname}.{tld} because we want to develop sites for international audiences, like here in Uruguay, where all of our tlds have our country code at the end. (.uy)
In the first release of dynamic sites, I have a redirect function that only supports HTTP GET, for now. The requirements said to retain the HTTP method as best as possible, however I suspect that GET is only really necessary.
def redirect(self, new_host, subdomain=None):
"""
Tries its best to preserve request protocol, port, path,
and query args. Only works with HTTP GET
"""
return HttpResponsePermanentRedirect('%s://%s%s%s%s%s' % (
self.request.is_secure() and 'https' or 'http',
(subdomain) and '%s.' % subdomain or '',
new_host,
(int(self.port) not in (80, 443)) and ':%s' % self.port or '',
urlquote(self.request.path),
(self.request.method == 'GET'
and len(self.request.GET) > 0)
and '?%s' % self.request.GET.urlencode() or ''
))
Lastly, thanks to http://effbot.org/zone/django-multihost.htm, we add a process_response handler to call patch_vary_headers and instruct the Django caching system that the cache keys must depend on the HTTP_HOST, instead of just the request path (default). [docs]
Step 4: Dynamically Loading urls.py, and modifying settings. TEMPLATE_DIRS by site
Here’s where the real power comes in. Now that we have our site identified, we need to add some logic to modify from where urls.py and templates are loaded.
To do this, I ripped out a function from djangotoolbox called make_tls_property which, from the pydoc, “Creates a class-wide instance property with a thread-specific value.”
In short, I’m able to dynamically modify values from settings.py on a per request basis without worrying about upsetting other threads.
You can see an example of this in dynamicsites/middleware.py
SITE_ID = settings.__class__.SITE_ID = make_tls_property() TEMPLATES_DIR = settings.__class__.TEMPLATES_DIR = make_tls_property() ... self.site = Site.objects.get(domain=domain) SITE_ID.value = self.site.pk
Notice, that by dynamically loading a site-specific urls.py, you can specify exactly which views will be used for which URL’s, giving you dynamic views per site.
Note: You must define the sites in the database (using the admin panel or some other means) *before* enabling the middleware. If you enable the middleware, and no sites are defined in the database, your django site will throw 404′s for all requests.
Step 5: Dynamically map development environment hostnames to sites
In my development environment, I use very short codenames for the sites I’m developing. Mostly so I can access the site via my browser’s address bar as fast as possible, but also so I have a strong visual indicator which environment I’m looking at, to avoid making painful mistakes like accidentally editing data in the production environment.
Compare the following:
http://www.sobre-comida.com.br/some/path
http://www.sc.dev:8000/some/path
We just need to add an extra ENV_HOSTNAMES setting and a little extra logic to the middleware layer to support different hostnames depending on the environment. Note you’ll need to add the correct magic to your settings.py or deployment scripts to set ENV_HOSTNAMES correctly for your local dev, staging, test environments. ENV_HOSTNAMES should not be used in production!
'cus.dev': 'corp-umbrella-site.com',
'af.dev': 'about-food.com',
'res.af.dev': 'restaurants.about-food.com',
'sc.dev': 'sobre-comida.com.br',
'ag.dev': 'about.gam.es'
Step 6: Context Processor
Well, one last point. Let’s add the current site as a variable available to all templates via a context processor.
def current_site(request):
return (settings.SITE_ID) and {'site': Site.objects.get_current()} or None
If you notice in the screenshot for the modified admin interface above, it prints the current site name in the header bar. This was easy to do using this new context processor and a simple template override:
Simple make a /templates/admin/base_site.html template as such:
{% extends "admin/base.html" %}
{% load i18n %}
{% block title %}{{ title }} | {{ site.name }} {{% trans 'Administration' %}{% endblock %}
{% block branding %}
<h1 id="site-name">{{ site.name }} {% trans 'Administration' %}</h1>
{% endblock %}
{% block nav-global %}{% endblock %}
Wrapping It All Up
I encourage you to grab the source and set up your own little test environment. I’d be very interested to hear back any results of your tests, if you find any bugs, or if you have any suggestions for improvements to the code. In the case of the latter, I prefer to receive code samples or a link to a forked project on Mercurial.
UYSRC
References
- Django-dynamicsites – the code we released with this article
- http://effbot.org/zone/django-multihost.htm – most helpful of all
- http://www.huyng.com/archives/franchising-running-multiple-sites-from-one-django-codebase-2/394/ – Nice article, however it relies on thread-unsafe environment variables and Apache config
- http://stackoverflow.com/questions/1007017/sites-framework-on-a-single-django-instance
- http://stackoverflow.com/questions/2223713/how-to-locally-test-djangos-sites-framework
- http://eikke.com/django-domain-redirect-middleware/
