Unit testing for Google App Engine with Python

I recently inherited a python project that used Google App Engine, and didn't have any unit tests setup, which is something I wanted to fix. My main aim was to get to the point where I could test it like I did any Django app I've built (this GAE project does not use Django) - basically I wanted to be able to do tests where I can:

  • Query the database to check what I expect to be in it is
  • Call URLs on the app to check that the external interface works as expected

There's a little documentation on getting started with local unit testing for Python, but it's not clear from that how to run the tests, and there's no help on how to call URLs on your app that I could see. Thus, I thought I'd document how I got setup:

Infrastructure

First get setup to run the tests. I'm assuming you already have the GAE SDK for Python installed. Next I installed the following:

  • Python nose - testing framework for python, can be installed with "easy_install nose".
  • Nose plugin for GAE - a plugin that adds support for GAE. I downloaded and installed from tarball.
  • webtest - a framework that makes it easy to test WSGI apps. Can be installed using "easy_install webtest".

That's all the pieces. Once installed we're ready to start writing our unittests.

Writing tests

Nose, if you've not used it before, will look through your project for unittests and run them. That said, I still like to have them in an easy to find place, so I create a "tests" module in the root of my GAE project director.

   % cd [path-to-my-project]
   % mkdir tests
   % touch tests/__init__.py

In the tests directory I create a python file for each test class, and in __init__.py you should import that that class. So, if you had a myfirsttest.py that had a test class in it called MyFirstTest then your __init__.py would look like:

   from myfirsttest import MyFirstTest

Now, what does that class look like? Well, lets write a test that will test loading the root path on our GAE server. The code for that would look a little like this:

    import unittest
    from webtest import TestApp

    from google.appengine.ext import db
    from google.appengine.ext import testbed

    from my_gae_app import application

    class MyFirstTest(unittest.TestCase):

        def setUp(self):
            self.testbed = testbed.Testbed()

            # Then activate the testbed, which prepares the service stubs for use.
            self.testbed.setup_env(app_id='my_gae_app_id')
            self.testbed.activate()

            # Next, declare which service stubs you want to use.
            self.testbed.init_datastore_v3_stub()

            # create a test server for us to prod
            self.testapp = TestApp(application)

        def tearDown(self):
            self.testbed.deactivate()

        def testFetchRootURL(self):
            result = self.testapp.get("/")
            self.assertEqual(result.status, "200 OK") 

That gives you some idea of how to structure the test. The import of application is the application you create as part of your GAE project. You should have some code somewher that looks like:

   application = webapp.WSGIApplication([ bunch of url handlers specified here])

   def main():
        util.run_wsgi_app(application)

   if __name__ == "__main__":
        main()  

That's the application you give to webtest to wrap.

Also note that you import your data models and start using them or db.GqlQuery etc. in your tests.

Having written all that, you then go to the root director of your project and run it like so:

    % cd [path-to-my-project]
    % nosetests --with-gae
    .
    ----------------------------------------------------------------------
    Ran 1 tests in 0.396s\r\n\r\n    OK

Hopefully this'll help you get started at least - it took me a lot of head scratching to get that far, so I hope it can be of use to others and save them that head scratching time :)

Quiz Buster 1.2

Despite working full out on our next major thing (more soon!), we slipped out another point update to Quiz Buster, bringing it up to 1.2:

  • Wikipedia entries now easier to read using mobile site
  • Improved reference selection for wikipedia links
  • Fixed an issue with text scaling
  • Added more stock questions

You can snag Quiz Buster 1.2 on the app store, and as ever those of you who have it already should get an update when you next check for them. 

Quiz Buster 1.1 update out

We were fortunate enough to have some great feedback on Quiz Buster 1.0, and we took as much as we could and put out a quick 1.1 update. Changes in the update include:

  • Question entry screen can be cleared easily
  • Can now delete saved questions
  • Answer text resizes to fit screen
  • No longer creates duplicates in saved questions
  • Wikipedia view scales web page to fit screen better
  • Prevent network issues showing up as answers unknown

Quiz Buster 1.1 is now live on the app store, and hopefully those of you who have it already should get an update when you next check for them.

If you've not bought it, then why not? At $0.99 it's got to be worth having the premier quiz question answering app on your phone? :)

Quiz Buster!

Today we're pleased to announce that our fun little question answering app, Quiz Buster has been released for the iPhone and iPod Touch. Quiz Buster makes use of the amazing powers of deduction of the True Knowledge question answering system, and tries to help you answer that tricky question when caught short in the pub quiz, or to help if you need to appear knowledgeable quickly at your next party :) Who was UK Prime Minister in 1976? How old is Barack Obama? How tall is Naomi Campbell? Let Quiz Buster help you out!

True Knowledge's AI can answer a fantastic range of things, and there's a bunch of sample questions built into the app. If you find something it can't answer, then you can always head over to True Knowledge's website and teach the system yourself.

Quiz Buster is a fun app, either to help you out of a stick situation, or just for asking random bits of trivia to pass the time or test your friends. It's a bargin at 59p on the app store right now!

Migrating themes from Xcode 3 to Xcode 4 developer previews

I've been trying out the Xcode 4 Developer Previews, and the first thing that hit me was the current DPs don't find your nice font/colour settings for the code editor - a trivial thing on one hand, but something most coders are very sensitive about :)

The issue is it's not just a file location change (that is also the case), but it's also a file format change, so you can't just move your old custom theme into place. Thankfully aktowns has provided a nice ruby script to convert the themes from old to new format. You can grab the script from here - I modified it only to make it run slightly more easily on a stock ruby environment.

So, to migrate your themes:

  1. Install the "plist" ruby gem with: sudo gem install plist
  2. Download the script, and know where it is :)
  3. Your custom Xcode 3 themes are saved in ~/Library/Application\\ Support/Xcode/Color\\ Themes, go find the file there that you want to migrate (it'll have a .xccolortheme extension)
  4. Run dvtcolorconvert.rb passing it the path to your theme file
  5. It'll create a file in the same directory containing the theme in the new format with a .dvtcolortheme extension
  6. Move the .dvtcolortheme file into ~/Library/Developer/Xcode/UserData/FontAndColorThemes
  7. Restart Xcode 4

And your theme should now appear in the Xcode preferences. I found that it wasn't perfect - the font size got confused - but with developer previews I'd expect a little inconsistency. That said it was a few seconds to correct for that, compared to migrating it all by hand!

Visualising software development projects

Last month I had great fun working on the Boardgame Remix Kit iPhone app, which kept me busy for a large chunk of November (if you've not got the Boardgame Remix Kit on at least one media format, what are you waiting for?! It's an excellent xmas gift for yourself or others!). To the untrained eye it might look like all I did was sit hunched over a computer all day, but in fact virtually I was a busy little bee, as demonstrated by this visualisation:

Boardgame Remix Kit iPhone app went from nothing to shipping product over the course of the month, as visualised with the wonderful (if slightly arcane in use) gource.

Software projects are living, evolving beasts, particularly large projects with many collaborators. Whilst to the lay person it just looks like a bunch of weird text with too much  punctuation, there's a lot of activity going on, with different people working on different parts of projects, and it's sometimes hard to articulate that to non-techies. Thankfully gource helps capture all that in a way that's easy to understand.

Whilst the Boardgame Remix Kit software repository was only worked on by me, if you look at some of the other examples out there for larger projects (like this one of the Sakai education platform) then you can begin to see how this kind of thing goes from pretty video to a very useful project management tool. If all you have is a file system view of the project you might not notice when someone adds or removes a hundred files, but when visualised this way, it's easy to spot. Gource even has a real time mode where you can have it running all the time showing you things as they happen.

It also is yet another reason, as if you needed one, for people to use revision control - without taking regular snapshots of your development process this sort of thing isn't possible.

Drawing on the iPad

Whilst I'm very glad the iPad and iPhone don't require a stylus, having tried to use apps like Penultimate to take notes, I sometimes do wish that I could have access to one, as drawing, and in particular free form writing, with my finger ends up even messier than it otherwise would.

I was really taken by Dan Provost's excellent solution to this - he converted a dry erase marker into a conductive stylus that would work on the iPad screen. In particular, he noted that whilst most styluses try to mimic a pen, the iPad doesn't offer that level of finesse, and in fact has the right affordance for drawing with a marker pen - something I thought was spot on.

Whilst I liked Dan's hacked up pen idea, I felt the execution could be improved upon slightly - Dan's marker, like most, was plastic, so he had to run a wire around the body of the pen, making it look and feel unlike a regular marker. Thankfully, not all markers are plastic, and I found a Pentel Pen permanent marker which comes with a metal case:

CRW_3236.jpg

First I removed the existing nib from the pen, and then left the pen in the garage for a week to dry out - the last thing you want to do is draw in permanent ink on your iPad! After I'd made sure it had dried, I used a multimeter and a paperclip to confirm that the inside of the pen was not insulated from the metal casing.

Like Dan I used conductive foam for the nib (easily found in electrical component stores like Maplin), but despite the pen having a metal case, the last inch or so of the pen toward the nib is actually plastic, so you need to find a way to connect your new nib to the body of the pen. For this I filled end of the pen with some tinfoil, before inserting a straightened paper clip into the pen - this sinks into the foam filler inside the pen that normally holds the ink and gets entangled with all the tin foil - leaving just a few millimetres of clip exposed ready for the nib.

Finally, I then rolled up some conductive foam and slid (ahem, forced) that over clip and into the case. The tip of the nib will have a tendency to unroll itself, so you'll probably need to glue it a little, though be careful not to use too much glue, as the foam will absorb it and you'll compromise the conductivity of the nib. I also found that the layer of print on the pen wasn't as conductive as I'd first hoped, so I did have to scuff it slightly with a file, not a huge amount, but just enough to create a few bare spots of metal where my fingers rest.

The result is a nice marker pen that works with your iPad that has a nice look and feel to it. Just don't confuse it for another permanent marker at any point...

Adding OAuth support to ASIHttpRequest

On various projects I've been using the excellent ASIHttpRequest library as a way to talk to the network from both Mac and iPhone applications. If you need to access web services from your app, ASIHttpRequest does all the heavy lifting for you, and leaves you to get on with writing more interesting things. It also has native support for some specific services like Amazon's S3. Like I say, it's a real boon.

One thing it lacks though is support for OAuth - which a lot of web services require these days. Back in the summer I made a stab at adding OAuth support to ASIHttpRequest, and got something working that was good enough for me, but not for general release. I've finally found time to try and fix that!

Over on github I've just put up a fork of ASIHttpRequest with OAuth support. Ideally I'd like to try and get this pulled back into the main tree, though I think it needs a bit more testing before that happens. That said, I've been using this code for a while, so please give it a try and let me know what you think (or fork my code, change it, and send me a pull request :).