The cloud – Dotcloud for Geodjango

I’ve been looking at cloud platforms for Django over the past few days. I want to have a look at my application performance and scalability but I have a feeling my shared host, Webfaction (an excellent service for the price), might not be up to the job. With that in mind I was delighted to see heroku now offers a python platform. Heroku offers a simple but powerful way to scale applications and very programmer friendly deployment – git push.

Heroku was initially a platform for Ruby on Rails now offering python, java and more. However that ruby focus is still very evident in the documentation – most instructions for add-ons only refer to installing gems. Anyway I was still able to continue my deployment and the CLI was intuitive and I felt like I could be very productive on the platform. The add-ons like New Relic and Blitz.io were also excellent to use. It feels a lot more mature than a lot of the other PaaS out there but unfortunately I eventually had to abandon it as it doesn’t offer support for the GEOS library required by Geodjango. I also had problems with xapian but there is a solr add-on which shouldn’t have been to hard to swap in.

I searched in vain for a while for a more python/django focused offering and almost gave up until I found Dotcloud. Like heroku, Dotcloud offers various stacks but their python stack/offering seems much more mature. PostGIS support is excellent – just a matter of adding db: type: postgis to the dotcloud.yaml and the GEOS library is installed as well. Again like heroku there is no support for xapian but I switched to solr easily. Django-haystack made this task a lot easier as there wasn’t much search backend specific changes to be made. Finally I hade to switch my cache provider from memcache to redis but again it was fairly seemless.

So now its off to New Relic for some performance and scalability analysis, and monitoring…..

https://rpm.newrelic.com/public/charts/zbgAXPPBVL

https://rpm.newrelic.com/public/charts/e2QGWD4LRnD

Advertisements
Posted in Uncategorized | Leave a comment

Android Test Driven Development

Test Driven Development on android turned out to be much more frustrating than I expected. My hopes were high as I had seen that Android provides ActivityUnitTestCase for “isolated testing of a single activity”. However these tests must be run on a emulator or device making them rather slow. In order to do TDD the time taken to run the relevant unit test needs to be on par with saving a file i.e. in the order of milliseconds. In fact when doing TDD on an IDE I prefer to run a test (and automatically save the file) rather than saving directly. This way I get two bits of feedback every time I make a change 1. Does it compile? 2. Does the junit pass? This mode of work proved impossible using ActivityUnitTestCase as each run of a test involved a project being built and transferred to the device/emulator before being run taking ~10 secs each time.

I hope Google has plans for a speedier way to do this on Android but in the mean time I looked for an alternative. Robolectric objectives seemed to match what I was looking for so I gave it a go.  It intercepts the loading of Android classes so you can run unit tests on your local JVM. It seems like a hacky solution to me but until google tackles the core problem I can’t see any alternatives.

Below is my test.


package test.ie.shizzle;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import ie.shizzle.R;
import ie.shizzle.media.PlayerActivity;
import ie.shizzle.ref.StaticStrings;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.net.URISyntaxException;

import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;

import android.os.Bundle;
import android.widget.ImageView;
import android.widget.TextView;
import android.widget.VideoView;

import com.xtremelabs.robolectric.Robolectric;
import com.xtremelabs.robolectric.RobolectricTestRunner;

@RunWith(RobolectricTestRunner.class)
public class PlayerTest {
	PlayerActivity activityUnderTest;
	Bundle bundle;
	String[] videoIDs = {"tG7cM5Yvhz4", "v_UyVmITiYQ"};
	String[] videoNames = {"song 1", "song 2"};
	String artistName = "bojo";
	String gdataResponse;
	/**
	 * 
	 * Creates an initial bundle to pass the activityUnderTests onCreate method
	 * Creates a mock of a response from gdata from gdataResponse.xml
	 *  
	 * @throws IOException
	 */
	@Before
	public void setup() throws IOException{
		activityUnderTest = new PlayerActivity();
		bundle = new Bundle();
		bundle.putStringArray(StaticStrings.KEY_VIDEO_IDS, videoIDs);
		bundle.putStringArray(StaticStrings.KEY_VIDEO_NAME, videoNames);
		bundle.putString(StaticStrings.KEY_ARTIST, artistName);
		File file =  new File(getClass().getResource("gdataResponse.xml").getPath());
		FileReader fileReader =  new FileReader(file);
		BufferedReader bufferedReader = new BufferedReader(fileReader);
		StringBuilder builder = new StringBuilder();
		while(bufferedReader.ready()){
			builder.append(bufferedReader.readLine());
		}
		gdataResponse = builder.toString();
		TestHttpResponse response = new TestHttpResponse(200, gdataResponse);
		Robolectric.addPendingHttpResponse(response);
		activityUnderTest.onCreate(bundle);
	}
	
	/**
	 * Tests that the artist and song are displayed correctly
	 * Tests all the views are attached to the activity
	 * 
	 * @throws IOException
	 * @throws URISyntaxException
	 */
	@Test
	public void testPlayingFirstSong() throws IOException, URISyntaxException{
		assertNotNull(activityUnderTest);
		assertEquals(((TextView)activityUnderTest.findViewById(R.id.artist)).getText(),"bojo - song 1");
		assertNotNull(((VideoView)activityUnderTest.findViewById(R.id.video)));
		assertNotNull(((ImageView)activityUnderTest.findViewById(R.id.echoNextLink)));
	}
	
}

Its not very complex but I immediately ran into problems (see stack trace below) because my activity makes a google api call and the robolectric TestHttpResponse hasn’t implemented HttpEntityStub.getContentEncoding.



java.lang.UnsupportedOperationException
	at com.xtremelabs.robolectric.tester.org.apache.http.HttpEntityStub.getContentEncoding(HttpEntityStub.java:31)
	at com.google.api.client.http.apache.ApacheHttpResponse.getContentEncoding(ApacheHttpResponse.java:56)
	at com.google.api.client.http.HttpResponse.(HttpResponse.java:109)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:361)
	at ie.shizzle.media.PlayerActivity.onFillUI(PlayerActivity.java:62)
        ....

Not to worry I simply extended TestHttpResponse myself and implemented the required methods:



package test.ie.shizzle.support;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import org.apache.http.Header;
import org.apache.http.HeaderElement;
import org.apache.http.HttpEntity;
import org.apache.http.ParseException;

import com.xtremelabs.robolectric.tester.org.apache.http.HttpEntityStub;

public class TestHttpResponse extends
		com.xtremelabs.robolectric.tester.org.apache.http.TestHttpResponse {
    private TestHttpEntity httpEntity = new TestHttpEntity();
    private String responseBody;
    
	public TestHttpResponse(int statusCode, String responseBody) {
		super(statusCode, responseBody);
		this.responseBody = responseBody;
	}
	
    @Override public HttpEntity getEntity() {
        return httpEntity;
    }

    public class TestHttpEntity extends HttpEntityStub {
        @Override public long getContentLength() {
            return responseBody.length();
        }

        @Override public InputStream getContent() throws IOException, IllegalStateException {
            return new ByteArrayInputStream(responseBody.getBytes());
        }

        @Override public void writeTo(OutputStream outputStream) throws IOException {
            outputStream.write(responseBody.getBytes());
        }

        @Override public void consumeContent() throws IOException {
        }
        
        @Override
        public Header getContentType() {
        	return new Header() {
				
				@Override
				public String getValue() {
					// TODO Auto-generated method stub
					return "application/atom+xml";
				}
				
				@Override
				public String getName() {
					// TODO Auto-generated method stub
					return null;
				}
				
				@Override
				public HeaderElement[] getElements() throws ParseException {
					// TODO Auto-generated method stub
					return null;
				}
			};
        }
        @Override
        public Header getContentEncoding() {
        	return new Header() {
				
				@Override
				public String getValue() {
					// TODO Auto-generated method stub
					return "gzip";
				}
				
				@Override
				public String getName() {
					// TODO Auto-generated method stub
					return null;
				}
				
				@Override
				public HeaderElement[] getElements() throws ParseException {
					// TODO Auto-generated method stub
					return null;
				}
			};
        }
    }

}

That didn’t take me long but it does make me worry about this approach. I get the feeling I will end up spending a lot of time implemented mock classes that have only been stubbed out by robolectric. It reminds me a lot of trying to do TDD in Enterprise Java before Spring came along and made unit testing really simple, fast and effective.

Posted in Uncategorized | Leave a comment

Android JSON Adapter

I am taking a break from server side development for a few days and doing some android development. So its back to java *sigh* and so simple things seem much harder. I was surprised to find that Android doesn’t have an adapter for json given that so many web apis provide this format. I considered violating DRY, recreating my Django models in java and writing some conversion code, but I am going to see how I get on leaving all the model definitions to Django.

To this end, I created this naive JSON adapter based on the build-in ArrayAdapter and SimpleCursorAdapter.



package ie.shizzle;

import ie.shizzle.util.Utils;

import java.util.Arrays;

import org.json.JSONObject;

import android.content.Context;
import android.net.Uri;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;
import android.widget.ArrayAdapter;
import android.widget.CursorAdapter;
import android.widget.ImageView;
import android.widget.TextView;

/**
 * Simple Adapter for JSON
 * 
 * Based on {@link ArrayAdapter} and {@link CursorAdapter}
 * 
 * @author dave
 *
 */
public class JSONArrayAdapter extends ArrayAdapter {
	private int mResource;
	private LayoutInflater mInflater;
	protected String[] mFrom;
    protected int[] mTo;
    
	public JSONArrayAdapter(Context context, int textViewResourceId, JSONObject[] objects, String[] from, int[]to) {
        super(context, textViewResourceId, 0, Arrays.asList(objects));
		mInflater = (LayoutInflater)context.getSystemService(Context.LAYOUT_INFLATER_SERVICE);
        mResource = textViewResourceId;
        mFrom = from;
        mTo = to;
    }

	
	
	@Override
	public View getView(int position, View convertView, ViewGroup parent) {
		View view;
        if (convertView == null) {
            view = mInflater.inflate(mResource, parent, false);
        } else {
            view = convertView;
        }		
        JSONObject item = (JSONObject) getItem(position);
        bindView(view, item);

        return view;
	}
	/**
	 * a la bindView of {@link CursorAdapter}
	 * TODO allow for more complex bindings
	 */
	public void bindView(View view, JSONObject json) {
        final int count = mTo.length;
        final String[] from = mFrom;
        final int[] to = mTo;

        for (int i = 0; i < count; i++) {
            final View v = view.findViewById(to[i]);
            if (v != null) {
                    String text = "";
					text = Utils.getText(json,from[i]);

                    if (v instanceof TextView) {
                        setViewText((TextView) v, text);
                    } else if (v instanceof ImageView) {
                        setViewImage((ImageView) v, text);
                    } else {
                        throw new IllegalStateException(v.getClass().getName() + " is not a " +
                                " view that can be bound by this JSONArrayAdapter");
                    }
            }
        }
    }
	
	
	public void setViewImage(ImageView v, String value) {
        try {
            v.setImageResource(Integer.parseInt(value));
        } catch (NumberFormatException nfe) {
            v.setImageURI(Uri.parse(value));
        }
    }
	
	public void setViewText(TextView v, String text) {
        v.setText(text);
    }

	
}
Posted in Uncategorized | Leave a comment

Dojo Dijit Interface

I need to create a complex web interface for an application I am working on. I know from past experience that Dojo has some very interesting widgets for creating Web 2.0 GUIs but its been a while since I’ve worked with Dojo and I’ve never integrated it with Django before. So here goes…

D0jo – Django Integration

The first issue that occurred to me was how to integrate Dojo into my Django templates in a consistent and manageable way. Luckily there is already a Django application for exactly this purpose. Dojango is an application that helps you to use Dojo within Django. I simply downloaded and extracted the latest release.

tar xzvf dojango-0.5.1.tar.gz

cd dojango-0.5.1

sudo python setup.py install

Then I just added dojango as an application in my settings.py:

INSTALLED_APPS = (
‘django.contrib.auth’,
‘django.contrib.contenttypes’,
‘django.contrib.sessions’,
‘django.contrib.sites’,
‘shizzle.event’,
‘django.contrib.admin’,
‘django.contrib.gis’,
‘haystack’,
‘tinymce’,
‘dojango’
)

Then just enable dojango urls as in step 2 and you can verify by loading the test page http://localhost:8000/dojango/test/ . Now I can add dojo features to my pages by extending dojango/base.html like in this example. I don’t even have to import the javascript (its done in base.html) and if I decide to change to using a custom dojo build I can simply change the dojango configuration rather than changing imports all over the place.

Dijit Layout with Dojango

So dojango allows me get started with dojo but how to lay out my application is my next question. I haven’t discovered any good tutorials on starting work on a complex web application with dijit so I suggest you start by taking one of the layout examples from the dojo website and integrate it with dojango. I tried starting from scratch but its too easy to do something stupid like forget to set the size of the body element and end up with a blank page (It seems the dijit.layout containers have size zero by default).

Anyway here is my initial page based on the first example from the dojo website the “my first page” from dojango:


{% extends \"dojango/base.html\" %}

{% block dojango_page_title %}Event Dashboard{% endblock %}
{% block dojango_header_extra %}
<style type=\"text/css\">
            html, body { width: 100%; height: 100%; margin: 0; }
</style>
<script type=\"text/javascript\">
    dojo.require(\"dijit.form.DateTextBox\");
    dojo.require(\"dijit.form.TimeTextBox\");
    dojo.require(\"dijit.layout.ContentPane\");
    dojo.require(\"dijit.layout.BorderContainer\");
    dojo.require(\"dijit.layout.TabContainer\");
    dojo.require(\"dijit.layout.AccordionContainer\");
    dojo.require(\"dijit.layout.SplitContainer\");
</script>
{% endblock %}
{% block dojango_content %}
 <div dojoType=\"dijit.layout.BorderContainer\" style=\"width: 100%; height: 100%;\">
            <div dojoType=\"dijit.layout.ContentPane\" region=\"top\">
                Title
            </div>
            <div dojoType=\"dijit.layout.ContentPane\" region=\"leading\">
                <input type=\"text\" dojoType=\"dijit.form.DateTextBox\" value=\"T09:00\" />
                <input type=\"text\" dojoType=\"dijit.form.TimeTextBox\" value=\"T09:00\" />
            </div>
            <div dojoType=\"dijit.layout.SplitContainer\" region=\"center\">
                <div dojoType=\"dijit.layout.ContentPane\" title=\"Feed Entries\" >
                    Feeds
                </div>
                <div dojoType=\"dijit.layout.ContentPane\" title=\"Related Entries\">
                    Related
                </div>
            </div>
            <div dojoType=\"dijit.layout.ContentPane\" region=\"trailing\">
                Broad search results:
                    vids, links, wikipedia
            </div>
            <div dojoType=\"dijit.layout.ContentPane\" region=\"bottom\">
                Preview
            </div>
        </div>
{% endblock %}

Which gives me a layout like this:

Posted in Dojo | 1 Comment

Temporal Expressions

I’ve taken a dive in Natural Language Processing over the last few weeks. My initial requirement was to extract temporal information from a bunch of related feeds for a side project of mine. I did some research online and bought the excellent Natural Language Processing with Python.

The book is about NLP in general but the examples are very focued on the NLTK. NLTK is an open source toolkit for NLP in Python. It’s a really interesting tool and quite easy (especially with the book) to get into. If you have any interest in NLP and are reasonably capable with python this is a great place to start.

I hope to make more use of NLP in my project but to begin with here is how I dealt with the initial, relatively straight forward, requirement.

With a little googling I discovered contrib module called timex. Timex has two functions; tag() identifies temporal expressions in a text and tags them with a TimeML tag and ground() which fills the val attribute of the tags with the value represented by the temporal expression.

>>> from nltk_contrib.timex import *

>>> content =”Belfast up and coming band Cashier No.9 are playing a free gig in the Mercantile this Saturday night at 9.30pm. The band are set to release their debut album To The Death of Fun, which is mixed by David Holmes, in March.”

>>> tag(content)
‘Belfast up and coming band Cashier No.9 are playing a free gig in the Mercantile <TIMEX2>this Saturday</TIMEX2> night at 9.30pm. The band are set to release their debut album To The Death of Fun, which is mixed by David Holmes, in March.’

As you can see from the example timex neatly tags relative temporal expressions like “this Saturday”. This is essential when trying to parse feeds as people usually talk about time relatively rather than specifiying an absolute date.

The ground method can then be used to convert this relative temporal expression into an absolute date like so:

>>> content = tag(content)

>>> print gmt()
2011-01-21 11:03:49.67

>>> ground(content,gmt())

‘Belfast up and coming band Cashier No.9 are playing a free gig in the Mercantile <TIMEX2 val=”2011-01-22″>this Saturday</TIMEX2> night at 9.30pm. The band are set to release their debut album To The Death of Fun, which is mixed by David Holmes, in March.’

The second parameter is is required in order to calculate the date. In this case I have just specified the current date/time (2011-01-21 11:03:49.67) but in the case of feeds the published date of the entry is probably most appropriate.

This is a great leap forward but I’m not totally happy with the results. There are many other common formats used in the feeds that aren’t handled well by timex. For example:

>>> tag(“january”)
‘january’
>>> tag(“January”)
‘January’
>>> tag(“13th January”)
’13th January’
>>> tag(“this January”)
‘<TIMEX2>this January</TIMEX2>’
>>> tag(“2011-01-21 11:03:49.67”)
‘<TIMEX2<TIMEX2>2011</TIMEX2>-01-21 11:03:49.67</TIMEX2>’
>>> tag(“2011-01-21”)
‘<TIMEX2>2011</TIMEX2>-01-21′
>>> tag(“13/1/2011”)
’13/1/2011’
>>> tag(“9:00pm 13/1/2011”)
‘9:00pm 13/1/2011’

I think all lot of these can be improved on relatively easily. So next post I should have some suggestions for how to improve on timex.

Posted in Uncategorized | Leave a comment

A better source of GIS data

I have been meaning to following up on my last post for months now but it’s been too much fun playing around with GIS data.

After getting a geodjango application working with CSO data in my last post, I rapidly got tired of the sparse data available. Then with a little googling I discovered that the OpenStreetMap data is available under the Creative Commons ShareAlike license.

OpenStreetMap is a great initiative to provide free geographical information. It’s like wikipedia and google maps combined. If you go to http://www.openstreetmap.org/ you get a interface much like google maps except it is editable allowing anyone who is interested to add or correct map data. If you have a geodjango application with an admin interface up and running you will recognise the map formatting as openstreetmaps with openlayers is the default widget in the django admin interface for geographic fields. Below is what the Pavillion Bar, Trinity college looks like in my admin interface:An image of the admin interface showing the Pav bar

The results are very good at least for Dublin. The vast majority of POI are accurately placed and coverage is also very good too. OpenStreetMap data is available for download in various formats from http://download.geofabrik.de/osm/ . Now I need to discover how to export my updated data back to OpenStreetMaps to support this great initiative.

Posted in Geographic | Leave a comment

Experiments with GeoDjango – Using CSO boundary data

I came across this video about Geodjango on youtube the other day. The functionality looked really impressive so I decided to give it a go. For the most part I followed the instructions on the django site but I wanted to use some data relevant to Ireland:

  1. The first variation from the tutorial came when I  tried to install a postgres DB. I am on Ubuntu 10.04 which supports postgres-8.4 but it seems postgres-8.3 is required to get the GIS functionality enabled (At least to work with geodjango). So I unistalled postgres-8.4
    sudo apt-get purge postgresql-8.4

    and added these lines

    deb http://archive.ubuntu.com/ubuntu/ karmic main restricted universe
    deb-src http://archive.ubuntu.com/ubuntu/ karmic main restricted universe

    to my /etc/apt/sources.list. Then just run the apt-get install command as described in the tutorial.

  2. Rather than use the dataset from the tutorial I wanted to something more relevant to my home country Ireland. The OSI have a lot of data available on their site but its all very expensive so I started with the city_towns data from the CSO. Its not exactly fit for purpose but at least its free for non commerical use. It comes in the same shp format as described in tutorial so I just had to modify the model to match the fields from the CSO file.
    1 from django.contrib.gis.db import models
    2
    3 class District(models.Model):
    4     edid = models.CharField(max_length=254)
    5     label = models.CharField(max_length=254)
    6     mploy = models.MultiPolygonField()
    7     objects = models.GeoManager()
    8
    9     def __unicode__(self):
    10         return self.label
  3. Finally the most time consuming part was to guess what SRS (Spatial Reference System) was used by the CSO/OSI in the city_towns.shp file. It wasn’t listed in the fields so I had to experiment. After a few attempts where all the towns ended up in the middle of the Atlantic I discovered the correct SRSID was 29902. The ESPG site isn’t exactly user friendly but it can help to find the correct ID. I adapted load.py like so
    1 import os
    2 from django.contrib.gis.utils import LayerMapping
    3 from models import District
    4
    5 world_mapping = {
    6     'edid' : 'USERID',
    7     'label' : 'label',
    8     'mploy' : 'MULTIPOLYGON',
    9 }
    10
    11 world_shp = os.path.abspath(os.path.join(os.path.dirname(__file__), 'data/city_towns.shp'))
    12
    13 def run(verbose=True):
    14     lm = LayerMapping(District, world_shp, world_mapping,
    15                           transform=True, encoding='iso-8859-1',source_srs=29902)
    16
    17     lm.save(strict=True, verbose=verbose)
    18

Results:

I was able to create a django model of the Irish cities and towns listed by the CSO. Making it possible to do interesting queries such as

District.objects.filter(mploy__contains=pnt)

i.e. in which district (town/city) is this pnt. Below is a screenshot from the admin site showing the region for Oldcastle, Co.Meath:

Geodjango Admin for Oldcastle, Co.Meath

Geodjango Admin for Oldcastle, Co.Meath

Posted in Geographic | Leave a comment