Category: CDRC

I’ve been based at the ESRC Consumer Data Research Centre, a multi-university (UCL/Liverpool/Leeds/Oxford) lab focused on research and provision of specialist UK consumer datasets, since 2015. One of my first outputs was to adapt DataShine, which I’d created in 2013 as part of a previous UCL project, to produce CDRC Maps – to map some of the open datasets we held, and aggregates of some of the more interesting socioeconomic datasets that we produced from the controlled collections.

CDRC Maps is an OpenLayers-based “slippy” (pan/zoomable) map website consisting of pre-rendered raster tiles of choropleth maps of consumer metrics, layered under another raster “context” layer containing roads and labels, and a mask which results in only building blocks being coloured by the underlying choropleth. It served its purpose of showing impactful, pretty and effective maps of our UK socioeconomic datasets, but being a raster based map, with billions of tiles sitting on one of our old servers, it has been showing its age for a while.

The modern web mapping toolstack has moved on, with the rise of powerful web browsers with fast vector rendering, responsive design for smartphones and tablets, and comprehensive GUI frameworks that elevate regular Javascript. CDRC’s requirements have evolved too, with a desire for map visualisation that includes downloadable snapshots, basic analytical functions and filters, rather than the simple view-only concept of CDRC Maps, and a need to embed the map in stories and dataset records, rather than only sitting standalone.

CDRC Maps has also long been hosted directly at UCL, on a local development server. CDRC is a data research centre not a technology centre and there is a desire for use to our server infrastructure for data primarily. The website has long been the most popular public website for CDRC and also is prone to usage spikes due to mass media often finding that maps are a quick way to illustrate a story – or be the story – compared with raw datasets that are less immediately accessible with media deadlines. It was clear that an external host for the sites itself, and ideally the data that powers the site, would be preferable.

To address this and bring CDRC Maps up to date with the new data platform, the centre commissioned Carto and Geolytix to produce CDRC Mapmaker during late 2020. The developers created a Node.js based website that uses the Vue templating framework. Mapbox GL JS 1 is used for the map controls/canvas and the vector tile rendering. The map framework has recently become non-open but there is an open fork, MapLibre, which we will take a look at in due course. The development toolchain has also been brought up to date with industry practice, with proper source code management, continuous integration, rapid development/testing on localhost, and deployment through GitHub.

Map config is in Javascript but this component is separated from the templating Vue/Javascript allowing configuration and setup of new maps to be discrete from the main code itself.

Data is completely separated from the code and there is no server-side processing element for the code. (We do also use an external service, Google Analytics, for our stats). The data is hosted on Carto’s data platform, where a number of datasets are loaded, and also a postcode lookup table. Carto is in fact built on PostgreSQL/PostGIS and provides a management GUI to allow these to be managed independently of the map code.

While the complete (albiet minified) code, config, fonts, images and stylesheets are less than 4MB, the datasets themselves use approximately 7GB of space on the Carto servers. Each geography used (MSOA, LSOA, OA, local authority) has four spatial data files, representing the unmasked choropleth along with three levels of clipping – urban extent (towns/cities), detailed urban (village level) and individual building blocks.

The application is structured around presenting two types of maps – metric maps (which show a various continuous variables associated with a particular dataset, sliced into groups) and classification maps which categorise areas into a single value (sometimes with a hierarchy of levels) and generally include a pen portrait description of the category.

We were delivered six functioning maps and I have gradually worked on extending the codebase and GUI functionality to encompass the wider variety of maps that were on CDRC Maps and that are listed in CDRC Data. Quirks of each additional map have actually meant minor changes to the code in each case to accommodate them, but I am hopeful now that the codebase is broad enough to allow for additional maps to be added in the future with minimal effort.

For this first release of Mapmaker, there are around 30 maps, covering CDRC classifications such as Consumer Vulnerability and the Internet User Classification (IUC), CDRC metric products such as Access to Healthy Assets and Hazards (AHAH) and Residential Mobility (Churn) and some popular government datasets like the Index of Multiple Deprivation (IMD), VOA building ages and Ofcom broadband speeds/availability.

Users can filter maps based on one or more classification categories or on multiple metric value ranges, and a PDF report can be easily produced with a view of the current map, a key and accompanying text and direct link. Clicking many of the maps will not only present the metrics or portrait, but include statistics on proportions in the current administrative area or a custom drawn region. The user interface is deliberately simple with standard pan/zoom controls, map selector, postcode search and layer toggles – that’s it. Planned development in the short term will include an even simpler UI to allow for easily embedding the map in CDRC Data and other CDRC data-led outputs.

CDRC Maps is currently still available for the limited number of maps that show datasets not included on CDRC Data, and it does have the advantage of a pure raster display meaning that some of our controlled datasets which require limited dissemination can be included in this way – on CDRC Mapmaker we would be delivering the dataset to the user’s browser, which is not ideal. Our plan is to de-brand CDRC Maps to provide a home, outside of the core CDRC output, for these legacy maps, in the same way that we have a GitHub repository storing some of our older datasets no longer on our main sites. CDRC is now nearly 8 years old and as the centre’s focus has been refined, not all our older assets have remained central to its mission, but for research reproducibility and historic linking purposes, it is important to preserve these.

We hope CDRC Mapmaker forms a useful visualisation tool for some of CDRC’s many data assets, and its filtering and reporting functionality allow CDRC’s data to be viewed and used in new ways.

Above: Areas of east and south-east London with more than 8% of inhabitants being originally from (from top to bottom) India (in East Ham), Lithuania (in Beckton) and Nigeria & Nepal (in Abbey Wood).

[Updated] Ever wondered why some branches of Tesco, the ubiquitous supermarket, have an American food section, while others have a Polish food chiller? Alternatively, it might have a catch-all “World Food” aisle, or it might not. The supermarket is, of course, catering to the local community. Immigrants to the UK do not uniformly spread out across the country, but tend to cluster in particular localities.

The latest map that I’ve published on CDRC Maps is a Country of Birth map, which attempts to summarise such communities in one view. It uses the same technique as Top Industry, it maps the most common country of birth (excluding the home nation) of residents in each small area, as of the 2011 Census. The purpose of the map is to identify and map the approximate extent of single-country communities within the UK. For example, to see how big London’s Chinatown is, or whether a Little Italy in the capital still exists.

This map reveals such communities although there is an important caveat when looking at it. I have set out below the rules I applied when constructing it, the most important of which is that only 8% of inhabitants need to share a single country of birth, for it to appear on the map. Bear in mind that, across the UK, 87% of people were born here. These people do not appear on the map, unless they are outside their home nation (and not at all if they are English).

There are a number of rules I have needed to apply to make this a map that tells an interesting story in a measured and fair way:

• I don’t map native births – the English-born people in England, Welsh-born in Wales, Northern-Irish born in Northern Ireland or Scottish-born in Scotland. There are almost no areas anywhere in the UK where people born in a single foreign-born country outnumber the native-born. If I did map such native births, then the map would be almost completely dominated by them, and would not tell much of a story.
• I also don’t map the English-born within the other home-nations, because the population of England is so much larger than in Scotland, Wales etc such that even the small percentage of them moving into the other home nations would dominate the map of Scotland/Wales/NI, if included.
• I only map a single-country foreign born area if at least 8% of local residents are from that country. This sounds like a low threshold and it is – if an area is coloured a particular colour, it might still have up to 92% of the local residents actually being native-born.
• The above rule means that some very multi-cultural areas don’t get mapped, because they have a large number of non-native residents, but these are split amongst various countries such that none reaches the 8% threshold.
• Necessarily, in the source data, some countries are combined together into regions, either for a whole region (e.g. Central America) or for other countries in a region (e.g. Other East Asia, not including China/Japan etc). This is how the underlying Census statistics are represented. This can have the effect of making a result (for a region) appear when it wouldn’t otherwise appear (for any country in the region). However the number of places where this happens is small so it does not overly bias the map.
• A slight quirk of the census results is that the Scotland and Northern Ireland chose to, based on their own sum populations, aggregate some of the smaller-UK-population countries in a different way. For example, Northern Ireland doesn’t break out “Other Old EU” (e.g. Belgium) and “Other New EU” (e.g. Bulgaria) into separate categories. The Somalian population in Scotland is not presented as a distinct statistic, but it is in NI (and England/Wales). Again, this only affects countries/regions with smaller UK populations so doesn’t overly distort the map.
• I don’t colour the map where it would be showing data for less than 10 people. This causes a most noticeable rationalisation of the map in Scotland, because the small areas here have a lower population (typically 125 instead of 250 people). This means Scotland’s country-of-birth diversity is a little underrepresented when compared with the other regions of the map.
• I’ve used colour hues and brightnesses in an ordered way, to group together continents and regions. Greens = UK nations, Olives = Old EU, Browns = New EU, Yellows = North America, Pinks = Central America, Blues = Africa, Purples = Oceania, Reds = Asia. There is no particular meaning to the colours picked beyond this, but be aware that the eye is naturally drawn to some colour hues more than others.
• If a second country of birth also scores over 8%, but with a smaller local population than the first, then this is shown in striped lines over the first, and labelled as such in the interactive key.

Have a look at the map, and mouse around to find the meaning for the current colour, or see the scrollable key on the right.

Why 8%? I found that dropping this threshold (I tried initially at 5%) results in a lot of “noise” on the map, where only two large families need to move to an area, for it to acquire their birth-country colour. Increasing this threshold (e.g. to 10%, which I tried) results in many of the interesting patterns disappearing.

Interesting, some famous “immigrant” areas of London virtually disappear on this map. Brixton and Hackney are still associated with the Jamaican communities moving there in the 1940s/50s, but, at 8% threshold they virtually disappear. Only at 5% is there a significant community pattern appear. Similarly, Wandsworth and Shepherds Bush are known for their Australian communities but these also almost vanish when moving from 5% to 8%. At a 5% threshold, Hackney and Islington show a “patchwork” effect of integrated multicultural communities of Irish, Turkish, Nigerian and Jamaican-born immigrants. These also disappear largely from the map at 8% threshold. Remnants of the Irish migration to Kentish Town are more obvious.

London remains a fascinating mix where people from many different countries have set up their home in neighbourhoods with established communities and retail that cater for them. While the UK’s other cities have “international” quarters too, none shows the diverse nature of these communities. Virtually every country in the key has a London neighbourhood. (N.B. Places where there are pockets of many nations in a small area in London, and elsewhere, often indicate a student population at a globally well-known university).

Away from London, the Scottish-origin communities in Corby and Blackpool stand out, while the Americans on military bases in East Anglia also dominate the map there. Luton has a Polish, Pakistani and Irish disapora.

As ever, I am mapping small-area statistics, not those for individual houses (I don’t have that information!) and the representation of a particular house on the map is indicative of the local area rather than each house itself. The addition of houses on CDRC Maps maps is intended to make the map more relatable to the population structure of towns and cities, but it can make the data more detailed than it actually is. The map also includes non-residential buildings – there’s no easy way to filter these with the open dataset used, and the great majority of buildings in the UK are residential.

[Update – See this excellent article written by CityLab on this map, which explains some of the above nuances in a better way than I attempted to.]

Below: There is a Little Italy, but it’s in Peterborough now.