boxes, whiskers, and violins

My experiments with the charting language d3 continue.  I'm looking at different ways to display book review statistics.  For example, I'm interested in how long books sit on the shelf between being bought, and being read.  It can be a long time; but how long?

A scatter plot shows all the data:

Here I have colour-coded book types as purple for non-fiction, green for science fiction, and orange for other fiction.  The existence of the line is clear: I can't read books before I acquire them.  (The green dot below the line in 1998 is an anomaly: either the acquisition date or the review date is wrong.  I have investigated, but can't determine which, so have left it as it is.)

The scatter plot shows all the data, and provides a visual clue that maybe the time between acquisition and reading isn't too long.  (Of course, this only shows the books I've acquired and read, not the ones I've acquired and are still languishing on the unread shelves!)

In order to better visualise the time delay, I plotted the data as box-and-whisker plots (showing median, quartiles, outliers, and here also the mean), overlayed with violin plots, (showing a more detailed estimate of the underlying distribution):

This was all relatively easy to do using the d3 chart library, and Jason Davies' science.js library for the kernel density estimator needed for the violin plot.

So from this I can see that I tend to read fiction almost as soon as I buy it, SF quite soon after, and that non-fiction sticks around on the shelves for longer.  This probably indicates that I buy non-fiction partly as an investment (it's my pension fund!).

Maybe I should spend more time reading them, and less time learning new languages so I can analyse how fast I'm reading them?

Read More

book review statistics in d3

My experiments with the charting language d3 continue.  Now that I can happily import csv files, I decided to have a go at visualising some data about the various book reviews on my website.

First, I charted the cumulative number of reviews, of the fiction, SF, and non-fiction books:

I was surprised at the total number of reviews, as I 'd never seen this all in one place before. And I was also surprised at the relatively steady rate of reading demonstrated.

I wrote a little python script to extract the figures from the database into a csv file.  I discovered that a few weeks of writing JavaScript seems to have over-written a large chunk of python syntax in my brain...

I also had a go at viewing the data according to date of publication, of acquisition, and of review.  I'm still playing around with those views, working out what they can show.  More experimentation is needed.

Read More

rainfall statistics in d3

My explorations of the d3 data visualisation library for JavaScript continue apace.  I have now added "box-and-whisker" plots to the 2012 rainfall, helping demonstrate the anomalously wet April and July, and dry August, last year.  This required calculating the various means, medians and quartiles, which wasn't particularly difficult, and has given me a feel for working with arrays.

Box and whisker plots show minimum, lower quartile, median, upper quartile, and maximum.
The mean value is overlayed as a blob.

This latest version replicates the previous chart displaying this information, which I did in Excel. (The bars are slightly different, because here I have included the 2012 data in the statistics, whereas I excluded it in the previous version.)

I think this d3 version looks much cleaner.  It certainly lets me fiddle with things at the programmatic level I prefer!

Read More

interaction with d3

My explorations of the d3 data visualisation library for JavaScript continue.  Last weekend I dipped my toe in the water, and drew some simple rainfall charts.  The code was clunky, and the data was hardwired, but the charts were pretty enough to convince me to continue.

And so this weekend I've explored some more.  I've modified the code to read in the data from a csv file, and then refactored like mad to make it more elegant.  The result of half a day's work was a web page indistinguishable (on the surface) from before: but now the charts, and the headings, are generated from the csv file data, so I can add more data each month, and the page will update without any code having to change.  So, a success.

Then I decided to add a little bit of interaction: changing the colour of a bar or spot on mouse-over, and showing the actual data value.  The d3 library is powerful enough that it was very easy to do.

in May 2007 we had 76mm of rain

in July 2009 we had 63mm of rain

So that's some tasks completed off my to-do list.  I've still got a long way to go.  But, as when I was learning Python, I'm finding StackOverflow an invaluable resource.

The actual web page is here, but it may change over time as I add more functionality.

Read More

my first d3

Way back when I was an undergraduate, I stumbled across a book called Curve Tracing, by Percival Frost.  The college library had a first edition, published in 1872.  Several years later, I came across a newer edition in a bookshop, and grabbed at it.

This beautiful little book has tons of curves defined by equations, and shown as graphs.

Plate IV from Curve Tracing

This was all originally done in the days before computers.  It's been a goal of mine to reimplement some of this work, in an interactive form, because many of these curves have parameters that affect their detailed shape.  So, I've been looking for a suitable tool.

Also, I'm interested in data visualisation (and have long been a disciple of Tufte).  I've tried to follow the "no chartjunk" ethos in my own work.  For example, to plot our rainfall statistics, I had to struggle with Excel charts to remove most of the garish "ink" provided by default. But it's still not perfect.  Additionally, I have some specific work I want to do with modified parallel coordinates, for which there is no existing library.  So, I've been looking for a suitable tool.

Last week a colleague of mine mentioned d3, a JavaScript library for Data-Driven Documents. I browsed the gallery for a while, and fell in love.  I spent yesterday playing around with evaluating it, on simple charts, to see what it could do.  This meant reading Scott Murray's d3 tutorial, and implementing a few simple charts to show the rainfall data. (Oh, and learning my first JavaScript.)

First off, I tried a simple bar chart, starting from Murray's tutorial example (when learning a new language, I usually find it easier to modify existing code than start from an empty file). After fiddling around to get the ordinal x-axis working, I got:

d3 chart of 2012 rainfall in mm

I think that looks suitably chartjunk-free and minimal.  It also has the nice feature of combining the actual numerical values into the bars, giving what Tufte calls both a macro-reading (the bars: gosh, July was wet!) and a micro-reading (the numbers: July had 113mm of rain) in one chart.

What's nice about d3 is the way the axes scale automatically.  Exactly the same code produced these charts (the only difference is the July data value: 13, 113, 233):

automatic axis scaling with changing data values

But, of course, we have several years of rainfall data.  A little more hacking coding gave me a grouped bar chart:

rainfall, June 2005 -- May 2013

Now it's possible to see how wet April 2012 was, compared to normal Aprils, even if it wasn't as wet as that July!  My implementation of this is a little kludgy, with too much hard-wiring, since my goal was to evaluate the capability, not (yet) learn the entire language; my next task is to code it more elegantly.

I've always found grouped bar charts rather cluttered, and so I wondered if there was a better way to show the data.  Rather than use some sort of surface plot, I decide to try a projection where the size of a spot is related to the amount of rainfall.  With a surprisingly small change to the code, the grouped bar chart metamorphosed into a "blob" chart:

(left): blob area proportional to monthly rainfall; (right) blob radius proportional to monthly rainfall

This enables comparisons in both dimensions (years, or months), depending on whether you view rows or columns.

So, based on a day's work, I'm very impressed with d3.  However, there are a lot more d3 facilities I need to get up to speed with before I can start my reFrost project in earnest:

• csv data import -- currently the data is hard-wired into the scripts (ugh)
• data manipulation -- to calculate medians and quartiles for box and whisker plots
• lines -- to draw graphs rather than charts
• maths -- to calculate the functions: and, presumably I'll need a lot more JavaScript
• interaction -- so parameter values can be chosen by the user
• transitions -- so the graphs will smoothly change as parameters are varied
• more -- stuff I don't know about yet, but will need

So, a way to go, but I think I might have identified the tools I need.

Read More

food for thought

More than half of the world's population lives inside this circle despite it "being mostly water and including the most sparsely populated country on earth (Mongolia)"

Read More

a normal April again

Last April was anomalously wet.  This year, things were back to "normal", rainfall-wise, if not temperature-wise.  Despite being much drier, 2013 is still the second wettest April since (our) records began, with a measured 14mm:

April rainfall, in mm

Read More

It truly is an excellent paper

A few days ago I received the following email (details redacted to protect the guilty):

This is from the editorial board office of Journal of [Totally Unrelated to My Research (TUMR). TUMR] is a peer-reviewed international research journal, devoted to supporting a global exchange of knowledge of [TUMR].
We found a paper you had published in “Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)”.
Title: An architecture for modelling emergence in CA-like systems
Author(s): Stepney Susan
It truly is an excellent paper that relates quite nicely to our journal.
To promote the communications of [TUMR] and broaden our journal’s global perspective, we cordially invite you to submit new research manuscripts to our journal before Nov. 29, 2012.
You can enjoy a registration discount if your paper is accepted.

The paper in question does exist. It is totally unrelated to the remit of the journal, however.

There are two big red flags in the email that there is something phishy.

First, no-one in the know says a paper has been

published in “Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)”

That rather unwieldy title is used only on the LNCS publisher Springer's website, from where I suspect it was copied verbatim.

Second, the claim that the paper "relates quite nicely to our journal" is not true, as could be spotted by glancing at the paper's content for a split second.

No-one did glance at it, however. This is just computer generated spam. I know what journal TUMR really is, so I can guess a crude bit of keyword matching glommed on to the word "architecture" or "engineering" in the abstract. These terms are used in the paper in a completely different context from that of TUMR.  Clearly the paper was found by a mindless web trawl.

The key phrase that flags what is really going on here is near the end of the email: "registration discount". TUMR is an Open Access journal, which means you have to pay to publish.  They are trying to flatter me into thinking that they like my work, in order to convince me that I should pay to publish in their journal.  Since they have clearly never even read my work, however, the result is not my feeling sufficiently flattered to publish, but sufficiently aggravated to blog. 

This approach is a distant cousin of the classic perfect prediction scam.  Spam a large population with a range of information (the classic example is stock market predictions).  It will be false for most targets, who will just bin it.  But, by chance alone, it will be true for some, who, not realising the extent of the scam, and the vast number of false hits, will be convinced by the truth of the claim in their case.  Here it is true for those few targets who will, by chance alone, have "an excellent paper that relates quite nicely to our journal".  The implication is that you have been carefully selected.  The actuality is somewhat different.

This is the first time I've seen the scam used in this way.  But I'm sure it won't be the last...

Read More

better use seaweed

As Neils Bohr is alleged to have said, “prediction is very difficult, especially about the future”.

My smartphone has a weather app on it that gives a forecast of the next six days. I glance at it occasionally, but in September, we had our annual “Away Day” at work, which has a significant outdoor component. So I was watching the forecast quite carefully. It said it was going to be sunny. Great! Next day it was predicting cloud, then it changed back to sun, then to rain. Umm. Well, on the day it was fine, fortunately.

But all that made me a bit suspicious of the app. I know longer range forecasts are essentially useless, with the weather being a chaotic system, but I thought 4 or 5 days out was now in the bounds of possibility. So I’ve been keeping data from my app for the last month or so.

click to see the icons

Each strip of 6 icons represents the forecast for a given day: the item near the top the forecast six days in the future, the one at the bottom from just the day before. There’s a lot of variation.

Here’s a graph summarising the data.

click to see the scales

 For each day I’ve summarised the six icon strip with three numbers

1. #symbols (light green): the number of different forecasts given over the six days: a consistent forecast would use one symbol, with more symbols showing greater indecision. Over the 32 days of data collection, it managed the perfect “one symbol” 4 times (12.5%). 
2. #changes (mid green): the number of times the forecast changed its mind. This minimum possible value is #symbols – 1; a higher figure indicates vacillation. 
3. end game (dark green): the number of days the final symbol stayed constant. The maximum possible is six: a perfect six day forecast (managed 12.5% of the time); the least reliable is one day (managed 15 times, or 47% of the time.) 

Not very impressive. The same app also gives minimum and maximum temperature estimates, with similar meanderings.

I think I’ll go back to the trusty classic “Seaweed dry, sunny sky. Seaweed wet, rain you'll get.” We can add more states, with the well-known joke: “Seaweed gone: wind so strong!” And to bring it fully up to date we simply add: “raining seaweed, weird indeed”. 

Read More

citation patterns

An academic's contribution to the community is made through publications.  And so we obsessively keep good records of what we have published, and neurotically check out if anyone has noticed.

To this end, I’ve been playing around with Google Scholar, tidying up my auto-generated publications list there. Although I still maintain my own publications page, this separate one is a useful resource, mainly because it includes citation counts and links.

Citation links are good for more than justegoboo and feeding neuroses, honestly! Citations indicate other authors who find my work relevant to them, so it's very likely that their work is interesting and relevant to my own research. It makes sense for me to check them out, and then also check out the other papers they reference, and also their citations, to see if there’s something interesting and relevant to me. Although it is a bit easy to get sucked into a never-ending death-spiral of link following: there’s just too much interesting stuff.

That's the beauty of Google Scholar.  it doesn't just have a list of my papers (a task I can do myself with relatively little effort); it also auto-generates links to citing papers (a task that would be ridiculously time-consuming to do myself).

More than that, it also provides neat little graphs of citations per year. These are quite interesting for publications that are several years old, where patterns have had time to emerge.

The typical pattern shows a peak a few years after publication, then a tail off into obscurity.

Slightly more gratifying is the pattern of a peak, followed by a drop, but then carrying on bumping along at about the same rate.

But the weirdest citation pattern must be to a paper written during my PhD, published nearly 30 years ago. It caused barely a ripple when published (I comfort myself that there was no Web in those days, and hence restricted access to it). But then there was a second (modest) peak of citations after about 15 years, followed by a drop-off. And then there was a third peak, starting more than two decades after publication.

Clearly, a paper ahead of its time!

Read More

FatFonts

Edward Tufte advocates using minimum ink to maximum effect in displaying quantitative information, and designing graphic displays to have both macro (distant) and micro (close-up) readings.

One technique he describes that fits his philosophy is the "stem and leaf" plot. Let’s say you have two sets of 50 exam marks as percentages, and you want to examine the data.  One way to see the spread is to plot the marks, maybe as a histogram. That shows the distribution, but loses the information of the specific scores.  The stem and leaf plot provides the best of both worlds: a graphical overview and close up values.  The "stem" shows the tens values, and the "leaves" show the remaining digit values.

stem and leaf plots, of 50 values generated randomly from a normal distribution, with:

 (a) mean 50, std dev 20; (b) mean 70, std dev 10

This plot looks like a histogram from a distance, but closer inspection shows that the bars are made from digits with meaning.  So we can easily see that in the first case, the marks range from 9 to 96, and in the second case, the marks range from 54 to 94. 

Stem and leaf plots can also be used to good effect for bus and train timetables. In this case the "stem" shows the hours, and the "leaves" show the minutes when the transport is due.  For example, Nottingham tram timetables use this approach.

You can immediately see, with a distant macro-reading, on weekdays the most trams are at rush hour, on Saturdays they are more evenly spread , and they are rather more infrequent out of hours and on Sundays.  Then you can look closer for a micro-reading, and see the actual tram times.  

These plots have been around for a long time.  The latest New Scientist (2 May 2012) reports an analogous approach for greyscale plots: "FatFonts", conceived and developed by Miguel Nacenta, Uta Hinrichs, and Sheelagh Carpendale.

the "Rotunda" FatFont

With a FatFont, the amount of ink used to print a digit is proportional to the value of the digit: so a 9 uses nine times the ink as a 1, for example.  Instead of plotting a greyscale pixel on a graphic, plot the actual value using the FatFont.  For more numerical resolution, simply use a second digit one-tenth the area of the first (and a third digit, one tenth smaller again, if needed, and so on, until resolution fails). From a distance, the macro-reading is a greyscale image; closer up, the micro-reading shows the digits, showing the numerical values. 

Here's an image of the terrain around Etna on Sicily, from the FatFonts galley page: 

A FatFont plot is to a greyscale image what a stem and leaf plot is to a histogram.

Typewriter art, ACSII art, and photo mosaics achieve a given picture (macro-reading) by using very many small characters or picture "tiles" chosen to have the right density or colour.  But with those techniques, the micro-readings are not usually of interest.  FatFonts, on the other hand, give a useful graphical tool.

It’s a really neat idea.

Read More

April showers

We have an official drought here in the south of England; ironic in a country famed for its rain.  Of course, that's part of the problem: why bother to store something that's in constant supply.  So it takes only two dry years in succession to cause a problem. However, recently we've also had a lot of very heavy rain, or so it seems.  Is it really much wetter than usual, or have we simply  forgotten how wet it usually is?

Last year was the driest since records began.  Well, since our records began in June 2005, when we started collecting rainfall statistics in our Cambridgeshire garden.  The annual rainfall (in mm) since then shows last year very dry, with 2010 on the dry side.

annual rainfall, in mm

Actually, it isn't the total rainfall over the year that's important for droughts, it's the rainfall over the winter.  Summer rain evaporates more; winter rain soaks in.  So here's the rainfall month by month, averaged since (our) records began:

average (mean and median) monthly rainfall, in mm

So the wettest month by far is August, closely followed by May: neither exactly winter months. Also, the driest month by far is April: so much for it being famed for "April showers".  Yet it is this April that has been so seemingly wet.

However, plotting both mean and median points out an anomaly in the April value.  Its mean is much higher than its median, indicating a skewed distribution.  Let's look at April in more detail:

April rainfall, in mm

So, for three years we detected no rain at all in April, and in 2010 only 1mm.  Two further years were around 10mm.  Then this year, 72mm!  That's some outlier.  Not only has it been wet this April, it's been significantly wetter than the average for any month, and it's been over three times wetter than all those other Aprils put together! 

So yes, it really has been wet. And it's not only our garden, and our short measurement span.  It has in fact been the wettest April in the UK for over a hundred years.  But it's still officially a drought.

Read More

spotting the placebo

A lovely letter in New Scientist (22 Jan 2001, p26):

From Phillip Gething

As part of my training to join a medical research ethics committee, I attended lectures on drug trials. We were told of a trial in which a patient spotted when her drugs had been switched, even though great efforts had been made to make the capsules appear identical. When asked by her GP why she thought the drugs had been changed, the patient replied, "I always put them down the loo. Until recently they always floated, now they always sink."

Fleet, Hampshire, UK

Being able to spot the placebo clearly wasn't the main problem here!

Read More