Tom Martin

Monitoring Varnish with Server Density

2011-04-04T00:00:00+01:00

Varnish is a high performance HTTP Cache and provides incredible performance for sites and applications. Server Density is a hosted server monitoring application, which deals with reporting, managing and alerting you to any server issues. Server Density has support metrics such as Apache, Nginx, MySQL but not Varnish.

Varnish support isn’t provided by default, so here’s a Server Density Varnish plugin which will provide loverly looking graphs like:

Usage

Usage gives you a good breakdown of cache hits vs cache misses.

Cache Percentage

Cache Percentage is a pretty important metric, the higher the value the more performance you’ll get out of Varnish.

Backends

Gives you a good breakdown of how you backends, import metrics to notice are backend_busy and backend_fail

Total Bytes

It’s probably the least useful day to day, but can let you know if you’ve just served a surge of traffic.

With any of these graphs, Server Density allows you to set alerts for any important metrics.

Installing the plugin is fairly straightforward, Varnish needs to be running with the management port enabled next you’ll just need to edit the plugin to provide the port and IP address, next you’ll need to setup the graph groups from within SD. Unfortunately there isn’t any way to preload a plugin with graphs or provide configuration options.

If you’ve got some improvements to make, please fork the plugin and send me a pull request.

Building a hack night

2011-03-29T00:00:00+01:00

I’ve often been envious of hack nights that happen in SF, NY and many other two letter american states. I’m also especially jealous of all of the great events that happen in London town, stuff like the Web Performance Group and the London Ruby Group.

So, I created HydraHack, a meetup for developers in Birmingham who want to hack on projects with other developers or just grab some time for working on their own code. It should be less about sitting for 40 minutes though a presentation and more about meeting other people and sharing ideas and code.

Begin

Picking a date was the first step, even before I had a name or any idea who should turn up. Pick a name, book a location, now I had a deadline. I fired a few emails off to some local devs for a sanity check “Yep, good idea.”

Justine helped me figure out the name. HydraHack

Promote

To get some interest, I threw up a static page running on Heroku with the details and got the domain. I added a Twitter button which broadcast the meetup to anyone that RSVPed. Although I had to push a new version of the site anytime anyone actually pressed it, was actually quicker than writing any real code.

I had a few local developers in my Twitter list so they all got an invite, there’s also a couple of local PHP and Ruby meetups, they were a great place to get some initial interest. To find people outside my regular network I used really ugly services like Twellow to find some other web folk, there’s quite a high noise rate but I stumbled across a few people I’d never met before.

I also put the event up on Laynrd which is becoming the best place to RSVP/ Track events. Not only is tied in to everyone’s social graph, it helped me find a bunch of similar events and more local developers to share the event with.

Profit!

Kindly Campaign Monitor agreed to sponsor the drinks for the first meetup. With the lure of free coffee we managed we got 16 developers crammed into Urban Coffee Company hacking away on various projects. I’d just like to say thanks again to everyone that turned up, I couldn’t of asked for a better start.

I’m busy organising the second meetup, tracking down developers, getting a sponsor and finding shiny free tech for people to play with. I have no idea how it will evolve, but I like making things fast and watching what happens.

Jekyll on Heroku using Sinatra

2011-01-16T00:00:00+00:00

This site runs on Jekyll, the static site generator written in Ruby. It’s running on Heroku and thanks to Sinatra, it can take advantage Heroku’s HTTP Caching library to give a huge increasing in performance.

Jekyll is a pretty amazing little gem, it allows you to quickly build a static site (the output is entirely HTML) it uses a great tempting engine and an easy to understand folder structure. It’s no wonder that this has made an excellent choice for a simple blog, simply drop a textfile in the /_posts regenerate your site and you’ve just made a new post.

Heroku an extremely easy to use Ruby host, you can use it for free to to deploy Ruby on Rails and Rack applications. For Rack, you just need to drop in the correct config config and off you go. But what about Jekyll? Heroku uses a Read-Only file system so you can’t use Jekyll to automatically generate static pages on Heroku, instead you need to run Jekyll before you deploy creating all your pages and then push the entire app (including rendered HTML) to Heroku.

A Jekyll site doesn’t really fit into either the Ruby on Rails or standard Rack app pattern, so if you simply push the site up, Heroku will reject the application because it’s not a Rack or Ruby or Rails app.

Gem to the rescue.

When my initial push failed I went around looking for a solution, there’s a Rack-Jeykll gem which will transform your standard Jeykll site into a Rack application.

Unfortunately the way the Rack-Jeykll gem works, it doesn’t seem possible to add any kind of HTTP headers within the app. What HTTP headers would you want to add you ask? Well, to increase application performance you typically want to allow browsers and any proxies to cache anything which won’t change for a while. Heroku also runs Varnish, a HTTP Cache in front of your application, this will any request straight out of the Varnish without even hitting your application. It can give it a pretty sizeable speed boost, especially when using the free plan.

Welcome Sinatra

After a little searching, it turns out you can use Sinatra with Jeykll to easily run your app.

You config.ru file just needs to require your app.

And your app just needs to require the Sinatra gem, and route any requests though to the correct file in your _site folder. This allows any cache control headers to be set so both browsers and Varnish will cache the resources to improve performance.

This setup allows you to use Jekyll to serve static files while still using Sinatra for any custom routes or 404 pages. This setup also allows me to use the the post slug as the URL and remove the date.

Monitoring Memcached with Server Density

2011-01-06T00:00:00+00:00

Server Density is a fantastic product for monitoring server performance, but by default Memcached it not a supported service, fortunately you can write plugins in Python to extend monitoring to any kind of metric.

For Memcached there was a pre-existing plugin written by chrisreid, but this depends on the python-memcached library, which as a total newcomer to Python and packaging for Red Hat I found pretty difficult to install.

So I tried the simplest thing which could possibly work, turns out you can telnet into the memcached port 11211 and ask for stats. So with a little knowledge of Python regex and lots of tinkering I present a Server Density Memcached plugin.

Install instructions are really easy to follow and hopefully you won’t have too much trouble getting it setup, if the server is running on another IP or port to the defaults (127.0.0.1:11211) you’ll need to hack the plugin as there’s no way for me to allow config in SD.

What’s great about any of the plugins you install for SD is that they all act the same as the native metrics, so if you want to get alerted when your Memcached misses grow larger than your hits you can create an alert just as easy as you would for a high load or memory.

If you’ve got some improvements to make, please fork the plugin and send me a pull request, the plugin source is released under the New BSD License.