Category: Mashups

The Tech City Map

Last week the government launched the Tech City map. It is not the first – there have been several previous maps, e.g. by the Guardian, of technology startups in the Old Street/Shoreditch area of London, but this is the highest profile one produced yet. I’m not sure whether a map itself is useful to the community but it serves perhaps to increase the identity and branding of the area – and awareness of it for the general public.

Incidentally, the area covered is also known as the Silicon Roundabout area, it often incorrectly referred to the Westminster set as “East London” (I would more term it as North-East inner-city London). I would consider more for the area around the Olympic Park to be true East London, i.e. east of Charing Cross, north of the river and east of the City – areas such as the East End, Stratford, Barking and West Ham.

As for the cartography, there has been some criticism by some people of the mass of straight blue lines, representing retweets of one company’s message by others. I prefer to see it as a piece of “art” showing the interconnectiveness of the organisations. No one’s trying to do serious quantitative impact work with this, it’s just for visual impact and identification of a new organically-created geographical area.

It’s nice also that the custom style functionality of Google Maps v3 API is being used – giving the streets a nice blue hue rather than the very familiar (and quite distracting) regular colours on Google Maps. Of course Google has recently introduced quite low limits (2500 map loads) for free usage of its custom style functionality, I wonder if this website eventually breaching such a limit if it gets hit with another blaze of publicity?

Perhaps the most exciting news that came out of last week’s announcements was that UCL, Imperial and Cisco are collaborating to build a new Future Cities research facility in the Tech City area, which would be focusing on urban modelling and technology. This is music to my ears – I’m not sure whether UCL CASA, the lab where I work, has plans ever have a presence there but it would be an amazing place to work. Maybe there will be a CASA East in the not too distant future?

Mashups Technical

WMS and SVG Input with Mapnik and Cairo

Post author By Oliver O'Brien
Post date 10 November 2011
No Comments on WMS and SVG Input with Mapnik and Cairo

A rather techy post from me, just before the Final Project Post for GEMMA (my current project at CASA) is submitted, summing the project and applications up. Why? I wanted to share with you two rather specialist bits of GEMMA that required quite a bit of head-scratching and trial-and-error, in the hope that, for some developer out there, they will be of use in their own work. I’m cross-posting this from the GEMMA project blog.

One of the core features of GEMMA that I have always desired is the ability to output the created map as a PDF. Not just any PDF, but a vector-based one – the idea that it will be razor-sharp when you print it out, rather than just looking like a screen grab. I had written a basic PDF creator, using Mapnik and Cairo, for OpenOrienteeringMap (OOM), an earlier side-project, and because the GEMMA project is about embracing and extending our existing technologies and knowledge at CASA, rather than reinventing the wheel, I was keen to utilise this code in GEMMA.

Most of the layers were quite straightforward – the two OpenStreetMap layers (background and feature) are very similar indeed to OOM, while the Markers and Data Collector layers were also reasonably easy to do – once I had imported (for the former) and hand-crafted (for the latter) suitable SVG images, so that they would stay looking nice when on the PDF. The trickiest layer was the MapTube layer. For the terms of this project, the MapTube imagery is not a vector layer, i.e. we are not using WFS. However I was still keen to include this layer in the PDF, so I turned to Richard Milton (the creator of MapTube) and discovered there is an WMS service that will stitch together the tiled images and serve them across the net. I could combine this requesting the WMS images on the GEMMA server (not the client!), converting them to temporary files, and then using a RasterSymbolizer in Mapnik 2, and an associated GDAL filetype.

The trickiest part was setting georeferencing information for the WMS image. Georeferencing is used to request the image, but it is also needed to position the image above or below the other Mapnik layers. Initially it looked like I would have to manually create a “worldfile”, but eventually I found a possibly undocumented Mapnik feature which allows manual specification of the bounding box.

I’ve not seen this done anywhere else before, although I presume people have just done it and not written it down on the web, so here’s my take, in Python.

First we get our WMS image. MAP_W and MAP_H are the size of the map area on the “sheet” in metres. We request it with a resolution of 5000 pixels per metre, which should produce a crisp looking image without stressing the server too much.

mb = map.envelope()
url = maptube_wms_path + "/?request=GetMap&service=WMS&version=1.3.0"
url = url + &format=image/png&crs=EPSG:900913"
url = url + "&width=" + str(int(MAP_W*5000)) 
url = url + "&height=" + str(int(MAP_H*5000))
url = url + "&bbox=" + str(mb.minx) + "," + str(mb.miny) + "," 
url = url + str(mb.maxx) + "," + str(mb.maxy)
url = url + "&layers=MAPID" + str(maptubeid)

furl = urllib2.urlopen(url, timeout=30)

Mapnik doesn’t work directly with images, but files, so we create a temporary file:

ftmp = tempfile.NamedTemporaryFile(suffix = '.png')
filename = ftmp.name
ftmp.write(furl.read())

Next we set up the layer and style. It’s nice that we can pass the opacity, set on the GEMMA website, straight into the layer in Mapnik.

			
style = mapnik.Style()
rule = mapnik.Rule()
rs = mapnik.RasterSymbolizer()
rs.opacity = opacity
rule.symbols.append(rs)
style.rules.append(rule)
map.append_style(filename,style)
lyr = mapnik.Layer(filename)

Here’s the key step, where we manually provide georeferencing information. epsg900913 is the usual Proj4 string for this coordinate reference system.

lyr.datasource = mapnik.Gdal(base='',file=filename, bbox=(mb.minx, mb.miny, mb.maxx, mb.maxy)) #Override GDAL
lyr.srs = epsg900913

Finally:

lyr.styles.append(filename)
map.layers.append(lyr)

I’m excited about one other piece of code in the PDF generation process, as again it involves jumping through some hoops, that are only lightly documented – adding an SVG “logo” – the SVG in this case being the GEMMA gerbil logo, that Steve (co-developer) created from Illustrator. Cairo does not allow native SVG import (only export) but you can use the RSVG Python package to pull this in. I’m being a bit lazy in hard-coding widths and scales here, because the logo never changes. There are more sophisticated calls, e.g. svg.props.width, that could be useful.

	
svg = rsvg.Handle(gemma_path + "/images/logo.svg")
ctx = cairo.Context(surface)
ctx.translate((MAP_WM+MAP_W)*S2P-ADORN_LOGO_SIZE, CONTENT_NM*S2P)
ctx.scale(0.062, 0.062)
svg.render_cairo(ctx)

Note that we are calling render_cairo, a function in RSVG, rather than a native Cairo function that we do for all the other layers in the PDF.

The screenshot above contains data from the OpenStreetMap project.

Mashups OpenLayers

Heatmaps are Simple with HTML 5 and Canvas

Post author By Oliver O'Brien
Post date 12 January 2011
1 Comment on Heatmaps are Simple with HTML 5 and Canvas

Cross-posted from my orienteering blog.

As a Saturday-lunchtime project, I have created a heatmap of where the 2700-odd geolocated orienteering races have been held in Great Britain in the last two years.

The heatmap quickly shows clusters around the main urban areas, where the population sizes supply participation for many local events being put on. Another major bright-spot is the Lake District extract on the right – which contains a large amount of high quality terrain for events. Other areas, such as the Cotswolds NW of Oxford, seem to be somewhat underused.

If you have a browser than can handle the HTML5 Canvas tag (i.e. not Internet Explorer!) you can view the heatmap here. Zoom into your local town or city to see if events have been held there – when zooming in, you’ll need to adjust the two sliders most of the way to the right, so that individual events show up. With the individual settings, a single, isolated event will have very little impact on the heatmap.

The heatmap was possible thanks to the excellent Heatmap library produced for OpenLayers by Bjoern Hoehrmann. The map is powered by OpenLayers, with an OpenStreetMap basemap. I’ve used a custom colour ramp, based on one supplied by Colorbrewer. The custom map adornments are supplied by MapBox.

Creating a heatmap like this is very easily, with just a few lines of Javascript needed to add objects in. The Heatmap library does the rest. See Bjoern’s example for the documentation.

Mashups OpenLayers OpenStreetMap

More Circles on a Map – Orienteering Fixtures

Post author By Oliver O'Brien
Post date 5 January 2011
No Comments on More Circles on a Map – Orienteering Fixtures

Five years ago, I created a mashup of forthcoming orienteering fixtures in Great Britain, as listed by the sport’s national governing body, British Orienteering, on its website. It was based on the Google Maps v2 API, and a regular scraping of the HTML on their website, and was a set of pins on a map, coloured by the number of weeks to the event. On clicking a pin, you got a popup balloon with details of the event, and a link to the organising club’s website. A postcode locator, based on data from the NPEMap project, was added, so you could focus on events in your local area. You could also filter out far away events.

A couple of years later, British Orienteering’s web developers added their own map to their website – Google Maps v2 API based, with pins coloured by the number of weeks to the event, and a popup balloon, a postcode search and distance filter etc etc… The Unique Selling Point of my fixtures map was lost.

So, when a rewrite of British Orienteering’s website just before Christmas broke my map, I took the opportunity to rewrite it, as a vacation project, using the technologies I’ve been using a lot in 2010 – namely OpenLayers, OpenStreetMap, OS OpenData and coloured vector circles. The map is bigger, brighter, and hopefully more useable than the official map and my previous version.

You can see the new map here – with a mass of dots representing forthcoming fixtures, and circles surrounding the “home” postcode, backed by OpenStreetMap, with the postcode locator based on CodePoint Open from Ordnance Survey OpenData. Only the locator uses a database, the rest of the webpage is constructed on-the-fly from a webpage regularly copied from the British Orienteering website.

Not Scarborough…

The map remains subject to the quality of the data entered on the corresponding list – there is some limited tidying up of the data, but it’s difficult to correct grid references that result in events being in the sea – there’s currently one in the Irish Sea, as the event registrant entered “GR” as the grid reference letters, and this just so happens to be the location of the GR myriad. There is still work to be done on my new map, such as spotting obvious errors like this, guessing locations where a grid reference isn’t supplied, and perhaps including Northern Ireland’s events.

Incidentally, my original orienteering web map, which inspired my fixtures map, was one showing orienteering maps, it was written way back in August 2004, using a Flash mapping package by Map Bureau, with dots superimposed on top of a map pinched from Wikipedia. We’ve come a long way.

Mashups

Mapping our Tweeting MPs with MapTube

Post author By Oliver O'Brien
Post date 16 December 2010
2 Comments on Mapping our Tweeting MPs with MapTube

One of the geometries available in MapTube shows the 2010 parliamentary constituencies. I’ve used to create a map, based on the list of MPs’ Twitter pages listed at Tweetminster and updated to reflect the boundary changes earlier this year. The map is clickable – choose a blue constituency, i.e. one which has a tweeting MP, and click on it to get a direct link to their Twitter page.

There’s more MPs on Twitter than you might think – over 200 currently, which is roughly one-third of all the MPs currently in parliament.

The map is OK on its own, but it’s real power comes when you can combine it with another political one, using MapTube’s map search functionality. You have to be careful when combining maps together, as maps with lots of colours will quickly become tricky to interpret when the colours, with varying translucencies, are laid on top of each other. However, with simple maps, it works quite well. There’s currently one showing which MPs have agreed to back an Early Day Motion banning Wild Animals in Circuses.

If you overlay both these maps together, you can interpolate the colours to quickly identify which MPs are on Twitter but (for example) have not agreed to back the motion, and then tweet them! In this example, the “haven’t signed” colour is red, and the “has Twitter” colour is blue, so the “target” list is purple.

The map doesn’t include Northern Ireland as we don’t have the constituency boundary geometries for those.

Geodemographics Mashups

OAC Groups on MapTube – A Demographic Map of the UK

Post author By Oliver O'Brien
Post date 3 December 2010
No Comments on OAC Groups on MapTube – A Demographic Map of the UK

The 21 Output Area Classification Groups is a updated version of one of MapTube’s most popular maps of the UK’s geodemographic. The original has had over 17,000 views and the was the first map added to the social mapping website which now has had almost 1,000 maps uploaded to it.

The Output Area Classification takes data from over 50 variables from the 2001 census, and clusters it into 7 supergroups, each subdivided into 2-4 groups to make a total of 21. Each group indicates broadly similar characteristics. Each of the 220,000 output areas in the postcode (typically representing ~300 people, or ~6 postcodes) is assigned to a group best on the “best fit” for that area’s population, bearing in mind only one group can be assigned, regardless of the diversity of the population there.

The new map is improved from the original in several ways:

The map is now broken up into the 21 groups, rather than the 7 top-level supergroups, revealing greater detail of the UK’s geodemographic.
Taking advantage of new technology built into the latest MapTube release, you can now click on any point on the map, and see the name of the supergroup, group (and subgroup) for that particular area in the resulting popup.
The colour scheme has been modified slightly – the groups in the “Countryside” supergroup are sublter shades of green, so the map is not visually dominated by bright green when viewed at a small scale.
Each supergroup has a distinct colour, as before, and their constituent groups vary by brightness – e.g. the “Prospering Suburbs” supergroup’s four groups are different shades of red.
Northern Ireland is now included.
The source data is downloadable as a CSV file – follow the “More Information” link on the key.

Here is a special link to the map, with the “Satellite Hybrid” Google Map layer selected, which provides a contextual overlay to the new OAC Groups choropleth map, and maximum opacity – I think this version is the most useful view.

The new map is also currently the Featured Map on the MapTube homepage. (Disclaimer: this is only because I have access to the Featured Map picker!)

Mashups

Tracking Flush Tracker or Why Is It Flowing Uphill?

Post author By Oliver O'Brien
Post date 22 November 2010
2 Comments on Tracking Flush Tracker or Why Is It Flowing Uphill?

Flush Tracker is an interesting map-based visualisation of a slightly taboo, and therefore amusing, subject. It’s based on Google Maps, and allows you to enter a postcode and elapsed time, and then “track” a toilet flush apparently move through the sewerage network, a blue line gradually extending along twists and turns, to the local treatment works. You can also view existing tracks, which generally appear to have just started their journey from “famous” places, such as 10 Downing Street, Buckingham Palace etc.

The tracks are intriguing, they often appear to be close to the road network, but not exactly on it, with a kink every few hundred metres, suggesting somehow the map is somehow aware of a real pipe network – information which has been traditionally locked away in the GIS databases of the various water companies – not even OS MasterMap has this data, as far as I am aware.

What’s actually happening is a clever bit of Javascript trickery. When you make a request, or load in a current request, the data that comes back to the browser is a simple lat/lon of the start and the end of the “journey”. The start point is based on a Google Maps geocode of a postcode you enter. The end point is a genuine location of the water treatment works that is geographically closest to you – even if it’s uphill, i.e. almost certainly not actually the one you are connected to. For example, west London postcodes generally result in a route moving uphill to the NW, even though London’s treatment works are located “downstream” to the east, at Beckton or Thamesmead.

The clever bit is showing the “route” as a realistic sequence of pipes rather than a straight line between the start and finish. This is achieved doing a Google Maps road routing request from the start to the finish, which sends back a list of coordinates, each corresponding to a turn or other road junction on the route. Twenty or so of these are then picked on a pseudo-random basis, and straight lines are drawn, in sequence, through the twenty turn-points and to the finish.

If you look carefully, you will see that a turn of the “pipe” route always occurs at a road junction, and that the overall route generally corresponds to the Google Maps suggested road route, with road curves and certain junctions missed out and replaced by straight lines.

So – not a sudden opening up of another network dataset by the water companies, but a nice bit of map “trickery” nonetheless.

Left: Flush Tracker route. Right: Google Maps road route.

Data Graphics Mashups

Boris Bikes – The Flows Are Coming

Post author By Oliver O'Brien
Post date 6 November 2010
3 Comments on Boris Bikes – The Flows Are Coming

Adrian Short (@adrianshort) sent an FOI request for flow data for the first million journeys on the Barclays Cycle Hire bike shares in London (the “Boris Bikes”). TfL responded with a test dataset of the first 99 journeys -from roughly 6-7am on 30 July – and a promise that the data for the next 999,901 are coming!

I’m working on an adaptation of my bike share visualisation to show these flows. It’s not possible to show a million lines on the screen at the same time, so consolidation, selection and filtering will be applied, but it certainly is possible to show the first 100 – click on the graphic for a full-size image:

I’m using colour to indicate the direction of travel (see the wheel.) I’ve also shown the flows to a couple of specific docks:

I’ll build out the visualisation with the full data set and release it soon.

As start and end timings are included in the data, I’m sure it is only a matter of time before someone builds a version with little animated bikes moving from the stations into the city during the rush-hour, similar to Matthew Somerville’s excellent real-time tube visualisation, for which the underlying data unfortunately got pulled.

Data Graphics Mashups Technical

Fewer Cities, More Cities

Some bad news and good news about the Bike Share visualisation.

The bad news – the operator behind the schemes in Paris, Seville, Vienna, Dublin, Brussels, Valencia and Toyama asked me to stop getting the current bike share data from their websites. Although I was just loading their webpages, “in practice you are extracting data from [the operator’s] databases and re-utilising it” and “[the] databases are protected under the harmonised sui generis database right, as provided under Directive 96/9/EC: chapter III article 7 (1) and (2).”

For these seven cities, you can still see a historical snapshot from last Monday, when the feeds were switched off, but not the live status, historical animation or trend graphs.

This is despite a quick search on the web revealing a six-month collection of data for one of the schemes (at four minute intervals), the resulting trends being shown at a conference; a better-service campaign website, again for one of the schemes, with regularly updated performance tables; and an iPhone app pulling in the data from numerous schemes run by the operator, amongst others.

Digital Urban also mentioned this in the context of Bike-o-Meter, which uses the aggregated data from my Bike Share maps.

Now for the good news – I’ve added in five more cities – Rennes, Bordeaux, Zaragoza, Mexico City and Rio de Janerio. Yay! The inclusion of Mexico City and Rio should hopefully counter some claims of an European/English-speaking bias! Mexico City’s scheme appears to be concentrated in one very affluent district of the metropolis, while Rio’s is based on the seafront south of the city, rather than in the main urban area.

Rennes is a particularly interesting example, more about that shortly.

[Update – turns out I’m not the first.]