Better testing for golang http handlers

I'm writing this up as it doesn't seem to be a common testing pattern for Go projects that I've seen, so might prove useful to someone somewhere as it did for me in a recent project.

One of the things that bugs me about the typical golang http server setup is that it relies on hidden globals. You typically write something like:

package main

import (
    "net/http"
    "fmt"
)

func myHandler(w http.ResponseWriter, r *http.Request) {
     fmt.Fprintf(w, "Hello, world!")
}

func main() {
     http.HandleFunc("/", myHandler)
     http.ListenAndServe(":8080", nil)
}

This is all lovely and simple, but there's some serious hidden work going on here. The bit that's always made me uncomfortable is that I set up all this state without any way to track it, which makes it very hard to test, particularly as the http library in golang doesn't allow for any introspection on the handlers you've set up. This means I need to write integration tests rather than unittests to have confidence that my URL handlers are set up correctly. The best I've seen done test wise normally with this setup is to test each handler function.

But there is a very easy solution to this, just it's not really considered something you'd ever do in the golang docs - they explicitly state no one would ever really do this. Clearly their attitude to testing is somewhat different to mine :)

The solution is in that nil parameter in the last line, which the golang documents state:

"ListenAndServe starts an HTTP server with a given address and handler. The handler is usually nil, which means to use DefaultServeMux."

That handler is a global variable, http.DefaultServeMux, which is the request multiplexer that takes the incoming requests, looks at the paths, and then works out which handler to call (including the default built in handlers if there's no match to return 404s etc.). This is all documented extrememly well in this article by Amit Saha, which I can highly recommend.

But you don't need to use the global, you can just instantiate your own multiplexer object and use that. If you do this suddenly your code stops using side effects to set up the http server and suddenly becomes a lot nicer to reason about and test.

package main

import (
    "net/http"
    "fmt"
)

func myHandler(w http.ResponseWriter, r *http.Request) {
     fmt.Fprintf(w, "Hello, world!")
}

func main() {
     mymux := http.NewServeMux()
     mymux.HandleFunc("/", myHandler)
     http.ListenAndServe(":8080", mymux)
}


The above is functionally the same as our first example, but no longer takes advantage of the hidden global state. This in itself may seem not to buy us much, but in reality you'll have lots of handlers to set up, and so your code can be made to look something more like:

func SetupMyHandlers() *http.ServeMux {
     mux := http.NewServeMux()

    // setup dynamic handlers
     mux.HandleFunc("/", MyIndexHandler)
     mux.HandleFunx("/login/", MyLoginHandler)
    // etc.

    // set up static handlers
     http.Handle("/static/", http.StripPrefix("/static/", http.FileServer(http.Dir("/static/"))))
    // etc.

     return mux
}

func main() {
     mymux := SetupMyHandlers()
     http.ListenAndServe(":8080", mymux)
}


At this point you can start using setupHandlers in your unit tests. Without this the common pattern I'd seen was:

package main

import (
    "net/http"
    "net/http/httptest"
    "testing"
)

func TestLoginHandler(t *testing.T) {

     r, err := http.NewRequest("GET", "/login", nil)
     if err != nil {
          t.Fatal(err)
     }
     w := httptest.NewRecorder()
     handler := http.HandlerFunc(MyLoginHandler)
     handler.ServeHTTP(w, r)

     resp := w.Result()

     if resp.StatusCode != http.StatusOK {
          t.Errorf("Unexpected status code %d", resp.StatusCode)
     }
}

Here you just wrap your specific handler function directly and call that in your tests. Which is very good for testing that the handler function works, but not so good for checking that someone hasn't botched the series of handler registration calls in your server. Instead, you can now change one line and get that additional coverage:

package main

import (
    "net/http"
    "net/http/httptest"
    "testing"
)

func TestLoginHandler(t *testing.T) {

     r, err := http.NewRequest("GET", "/login", nil)
     if err != nil {
          t.Fatal(err)
     }
     w := httptest.NewRecorder()
     handler := SetupMyHandlers()  // <---- this is the change :)
     handler.ServeHTTP(w, r)

     resp := w.Result()

     if resp.StatusCode != http.StatusOK {
          t.Errorf("Unexpected status code %d", resp.StatusCode)
     }
}

Same test as before, but now I'm checking the actual multiplexer used by the HTTP server works too, without having to write an integration test for that. Technically if someone forgets to pass the multiplexer to the server then that will not be picked up by my unit tests, so they're not perfect; but that's a single line mistake and all your URL handlers won't work, so I'm less concerned about that being not picked up by the developer than someone forgetting one handler in dozens. You also will automatically be testing any new http wrapper functions people insert into the chain. This could be a mixed blessing perhaps, but I'd argue it's better to make sure the wrappers are test friendly than have less overall coverage.

The other win of this approach is you can also unittest that your static content is is being mapped correctly, which you can't do using the common approach. You can happily test that requests to the static path I set up in SetupMyHandlers returns something sensible. Again, that may seem more like an integration style test, rather than a unit test, but if I add a unit test to check that then I'm more likely to find a fix bugs earlier in the dev cycle, rather than wasting time waiting for CI to pick up my mistake.

In general, if you have global state, you have a testing problem, so I'm surprised this approach isn't more common. It's hardly any code complexity increase to do what I suggest, but your test coverage grows a lot as a result.

Why I moved to Hockey App

Until recently, if you were testing iOS apps, there was one third party service that was absolutely essential - TestFlight. TestFlight is a web service that helps you manage all the apps you develop and your app testers, and makes it easy for testers to install your apps on their iOS devices (something Apple have made quite tricky otherwise). Over time TestFlight added more and more functionality, making it easy to retrieve crash logs, add checkpoints to see which features were tested and which weren't, and provide reminders to testers when new versions are available. TestFlight was a godsend to both developers and testers alike. And on top of everything, it is totally free - how much better can you get?

However, I recently came to find TestFlight wanting, and switched to an alternative paid for service, HockeyApp. At a fundamental level HockeyApp does a fairly similar set of things to TestFlight, but is a little less slick than TestFlight, and costs money. A few people have asked why I switched, so here's a quick summary of where HockeyApp wins over TestFlight for me.

Multiple platform support

TestFlight only supports iOS, but of late I've been working on a couple of things for OS X, and HockeyApp supports testing OS X apps in addition to iOS (it also claims to support Android development, but that's not something I've had cause to investigate yet). In general, testing on the Mac is easier than testing on iOS, as Apple have no special technologies in place to limit how you distribute test applications on the Mac, but even here HockeyApp make the process much better with their help than without.

Sparkle is an all but ubiquitous open source library for rolling out app updates on the Mac, or at least it as before the Mac App Store took over that duty. Sparkle will sit in your app and monitor a appcast feed from a server (much like an RSS Feed) to check for updates. HockeyApp supports using Sparkle - when you upload test builds they'll appear in a private appcast feed, and if you tell your app to look at that all your users will get notified the moment there's a new update, ensuring everyone is up to date.

Even if you intend to ship through the Mac App Store, which doesn't allow you to use Sparkle, you can use Sparkle during testing and then remove it when you submit the app to Apple - it's just a huge time saver.

And needless to say, that Hockey App supports both iOS and Mac means I can just use one service rather than two, which is always good.

Uptime

After almost a year of flawless service, TestFlight had some unfortunate downtime recently. Running a web service is difficult, so I appreciate that sometimes things go down, and they have my sympathy there. But I hit a situation where TestFlight went down twice for the better part of a UK working day (they're a US company, so they're most likely asleep at that point), and during the second event there was no acknowledgement that I could see from TestFlight that anything had gone wrong, and during that time I couldn't get important test builds out to my clients, leading me to have to apologise to people for something beyond my control.

This is where the \"free\" suddenly is less appealing. I don't really feel I can complain when something offered for free goes away. It's not that you get what you pay for, but it's more that there's no way in which TestFlight are particularly beholden to users when the users don't pay.

HockeyApp do charge, which makes the relationship much easier to understand. HockeyApp's uptime has been thus far been very good, but of course all web services will have times when someone goes wrong beyond anyone's control, so I don't expect HockeyApp to be above the occasional outage. But I feel they'll take it more seriously given that I'm paying for the service.

Integration with Xcode

I gather TestFlight does something similar now, but at the time when I switched I still had to upload apps to TestFlight by hand via a browser upload. HockeyApp have a little app for the Mac that you can integrate with Xcode as part of the app archive process, which will upload new builds to HockeyApp directly. Combining this with automatically generating version numbers from git means I now don't need to lift a finger when getting new test version of my apps to testers.

Nothing is prefect

This all makes it seem like HockeyApp must clearly be the way and anyone that uses TestFlight is in the wrong, but we must give TestFlight its due here - there's still some things that TestFlight do better than HockeyApp. TestFlight's website is just generally a little more slick than HockeyApp - be it setting up new apps, adding users, and so forth. TestFlight's library for getting crash logs, adding checkpoints from apps, and grabbing new builds is just a single library - you need to add two for HockeyApp and it's just a little bit more involved. But none of this adds up to much more friction in the long run, but it might be enough to keep some people from switching.

I started trying out HockeyApp just for my Mac OS X projects, but at the end of the day I switched wholeheartedly when I was unfortunate that their downtime coincided with a critical time in my app development and was unable to reach my testers twice in a month. I suspect that's atypical, and you should have a look at both services for yourself, but I think it shows that is just how important to developers apps like TestFlight and HockeyApp are now - they're essential infrastructure for developers, and we're severally impacted when they go away. I'm happy to pay for a service like this in order to try to ensure it's reliable - I just can't work without it.

Update: I got a nice note from the people at HockeyApp letting me know that they'll be moving to having just a single library for managing in app updates and crash logging etc., which is great news.

Unit testing for Google App Engine with Python

I recently inherited a python project that used Google App Engine, and didn't have any unit tests setup, which is something I wanted to fix. My main aim was to get to the point where I could test it like I did any Django app I've built (this GAE project does not use Django) - basically I wanted to be able to do tests where I can:

  • Query the database to check what I expect to be in it is
  • Call URLs on the app to check that the external interface works as expected

There's a little documentation on getting started with local unit testing for Python, but it's not clear from that how to run the tests, and there's no help on how to call URLs on your app that I could see. Thus, I thought I'd document how I got setup:

Infrastructure

First get setup to run the tests. I'm assuming you already have the GAE SDK for Python installed. Next I installed the following:

  • Python nose - testing framework for python, can be installed with "easy_install nose".
  • Nose plugin for GAE - a plugin that adds support for GAE. I downloaded and installed from tarball.
  • webtest - a framework that makes it easy to test WSGI apps. Can be installed using "easy_install webtest".

That's all the pieces. Once installed we're ready to start writing our unittests.

Writing tests

Nose, if you've not used it before, will look through your project for unittests and run them. That said, I still like to have them in an easy to find place, so I create a "tests" module in the root of my GAE project director.

   % cd [path-to-my-project]
   % mkdir tests
   % touch tests/__init__.py

In the tests directory I create a python file for each test class, and in __init__.py you should import that that class. So, if you had a myfirsttest.py that had a test class in it called MyFirstTest then your __init__.py would look like:

   from myfirsttest import MyFirstTest

Now, what does that class look like? Well, lets write a test that will test loading the root path on our GAE server. The code for that would look a little like this:

    import unittest
    from webtest import TestApp

    from google.appengine.ext import db
    from google.appengine.ext import testbed

    from my_gae_app import application

    class MyFirstTest(unittest.TestCase):

        def setUp(self):
            self.testbed = testbed.Testbed()

            # Then activate the testbed, which prepares the service stubs for use.
            self.testbed.setup_env(app_id='my_gae_app_id')
            self.testbed.activate()

            # Next, declare which service stubs you want to use.
            self.testbed.init_datastore_v3_stub()

            # create a test server for us to prod
            self.testapp = TestApp(application)

        def tearDown(self):
            self.testbed.deactivate()

        def testFetchRootURL(self):
            result = self.testapp.get("/")
            self.assertEqual(result.status, "200 OK") 

That gives you some idea of how to structure the test. The import of application is the application you create as part of your GAE project. You should have some code somewher that looks like:

   application = webapp.WSGIApplication([ bunch of url handlers specified here])

   def main():
        util.run_wsgi_app(application)

   if __name__ == "__main__":
        main()  

That's the application you give to webtest to wrap.

Also note that you import your data models and start using them or db.GqlQuery etc. in your tests.

Having written all that, you then go to the root director of your project and run it like so:

    % cd [path-to-my-project]
    % nosetests --with-gae
    .
    ----------------------------------------------------------------------
    Ran 1 tests in 0.396s\r\n\r\n    OK

Hopefully this'll help you get started at least - it took me a lot of head scratching to get that far, so I hope it can be of use to others and save them that head scratching time :)