We currently rely on varnish to serve up our posts and other pages which are largely static.In fact 40% of requests to our site never hit our web servers as they are served out of varnish’s cache first.For redundancy and also to scale varnish we run two instances(soon to be 3).We initially used varnish’s hashing algorithm based on uri, this worked fine and specific pages were only stored on one varnish.The problem we ran into was when we had to purge a page we ended up sending the command to both varnishes. This causes several problems one is which it simple isn’t scalable, image sending the same purge command to 10+ instances.Another problem with it was the size of the purge.list was twice what is should be. If you manage varnish you know that when the purge list gets too big varnish stops working.
What we decided to do was to direct requests based on the first character of the host name. This works for us because each user has their own subdomain name. Now ont only do all pages for a single user exist on one varnish instance but we can accurately direct the purge request too. One small note if you are going to do this based on uri instead of hostname you will need to edit the regular expression to use the second character as the first will always be a forward slash. In our config below you can see that all hostnames starting with 0-9 & a-j live on web11, everything else lives on web12, in the case of posterous.com requests(which are not cachable we round robin between the two webs. The haproxy config for this is below:
backend posterous_http_web11mode httpserver web11web11:80 checkbackend posterous_http_web12mode httpserver web12web12:80 checkbackend posterous_http_allmode httpserver web11web11:80 check server web12web12:80 checkfrontend posterous_httpdoor 172.27.0.20:80acl posterous_com_root hdr_beg(host) -i posterous.comacl a-q_hostnames hdr_beg(host) -i 0 1 2 3 4 5 6 7 8 9 a b c d e f g h i juse_backend posterous_http_all if posterous_com_rootuse_backend posterous_http_web11 if a-q_hostnames default_backend posterous_http_web12
dodizzle 