INTERACTIVE: TweetPolling Public Opinion

So we’ve all seen the varying opinion polls predicting what will happen in next year’s Scottish referendum on independence, but none of these polls show a running live feed of thought on the issue in the run-up to 18 September 2014.

Nor do any of them tap into what people are thinking/saying on social media — of particular importance when the voting age has recently been lowered to tech-savvy 16-year-olds.

And none of them cross reference opinion with census information i.e. immigrant population and gender/age rations of wards or constituencies in Scotland’s towns and cities.

Imagine a tool then that could do just that. A tool that could give you a real-time snapshot of thought in your town. A tool that could show you how the yes/no vote fluctuates over time following events and speeches in the countdown to the day of the vote.

So I thought it’d be an interesting exercise to develop a tool that can in theory do all these things. Below is an initial prototype – it’s still a tad rough around the edges of just the initial tweet-mapping functionality for Glasgow, but it gives you an idea of how the media could use this kind of tool. I still have to (a) add other towns/cities, (b) add a section to show all people’s comments, (c) add a cross-reference section of census info per ward, and (d) create a TweetBot to do load- and stress-testing on the tool.

But hey – baby steps.

So how to get the data? Well, this was the ‘easy’ part. Using the ScraperWiki platform I created a webscraper written in Python that, using Twitter’s own search API, searched for unique hashtags. The code would be looking for tweets in this kind of format “Screw the English! Yes #gla-07”. Now the comment wouldn’t be mandatory (although I still save that too) but I’m essentially mainly interested in the yes/no vote and what ward you live in (the hashtag) – and there’s a check so that you can only ever vote once. All this data is saved to an SQL database.

I’ve scheduled the code to run once an hour to ensure a staggered live feed.

Then it got interesting – mapping the damn data.

After a lot of research I decided to use javascript and the mobile-friendly Leaflet library. My javascript reads the results of an SQL query on the database – a query that does a count of yes/no votes for each ward. The result of this query is read via a JSON feed. My javascript then cycles through each row in the dataset and formats it in a GeoJSON which Leaflet needs for the mapping process.

One other step was to find the map coordinates of Glasgow’s wards. These I luckily found on the Glasgow City’s wonderful Open Data portal. Using the QGIS freeware I converted their .shp file into a GeoJSON file. These coordinates were then also passed to Leaflet to map the results to a ward.

There you have it. Easy.