Unconventional Computation with Evernote

Today I'm off to the conference on Unconventional Computation and Natural Computation (UCNC) in Milan.  I'm running the CoSMoS (Complex Systems Modelling and Simulation) workshop there on the Monday, including giving a talk.  Then it's several days of listening to lots of interesting talks from the "here be dragons" end of computing.  On the Friday, one of my students is giving a couple of talks on his work.

This is the first conference I've been to where I'll be taking notes in Evernote on my Netbook, rather than with a pen in a lab book.  Last year, UCNC was in September, and I didn't start using Evernote until October.  I did use Evernote at the science fiction convention at Easter, where it worked really well, but this is the acid test: using it for the day job!  Like at the con, I've set up a bunch of template notes, one for each session, ready to start taking notes.

I hope there is somewhere to rest the Samsung Netbook so its sharp little feet don't bruise my knees like they did at the con!

Read More

boxes, whiskers, and violins

My experiments with the charting language d3 continue.  I'm looking at different ways to display book review statistics.  For example, I'm interested in how long books sit on the shelf between being bought, and being read.  It can be a long time; but how long?

A scatter plot shows all the data:

Here I have colour-coded book types as purple for non-fiction, green for science fiction, and orange for other fiction.  The existence of the line is clear: I can't read books before I acquire them.  (The green dot below the line in 1998 is an anomaly: either the acquisition date or the review date is wrong.  I have investigated, but can't determine which, so have left it as it is.)

The scatter plot shows all the data, and provides a visual clue that maybe the time between acquisition and reading isn't too long.  (Of course, this only shows the books I've acquired and read, not the ones I've acquired and are still languishing on the unread shelves!)

In order to better visualise the time delay, I plotted the data as box-and-whisker plots (showing median, quartiles, outliers, and here also the mean), overlayed with violin plots, (showing a more detailed estimate of the underlying distribution):

This was all relatively easy to do using the d3 chart library, and Jason Davies' science.js library for the kernel density estimator needed for the violin plot.

So from this I can see that I tend to read fiction almost as soon as I buy it, SF quite soon after, and that non-fiction sticks around on the shelves for longer.  This probably indicates that I buy non-fiction partly as an investment (it's my pension fund!).

Maybe I should spend more time reading them, and less time learning new languages so I can analyse how fast I'm reading them?

Read More

book review statistics in d3

My experiments with the charting language d3 continue.  Now that I can happily import csv files, I decided to have a go at visualising some data about the various book reviews on my website.

First, I charted the cumulative number of reviews, of the fiction, SF, and non-fiction books:

I was surprised at the total number of reviews, as I 'd never seen this all in one place before. And I was also surprised at the relatively steady rate of reading demonstrated.

I wrote a little python script to extract the figures from the database into a csv file.  I discovered that a few weeks of writing JavaScript seems to have over-written a large chunk of python syntax in my brain...

I also had a go at viewing the data according to date of publication, of acquisition, and of review.  I'm still playing around with those views, working out what they can show.  More experimentation is needed.

Read More

sequestering carbon, several books at a time V

This week's haul, from Amazon and from the Europa 2013 archaeology meeting:

I've nearly finished the Kontis.

Read More

Iain M. Banks, 1954-2013

Iain Menzies Banks

I remember during the 1987 Worldcon hearing the rumour that Iain M. Banks had climbed up the outside of the Brighton Metropole hotel the previous night, to get to his room.  He told the story himself at the 1990 Eastercon in Liverpool, during his Guest of Honour talk.  At the 1998 Eastercon, Iain M Banks and his alter ego Iain Banks took to the stage to discuss SF versus mainstream writing.  He told wonderful stories, both in person and in print.

In early April this year, he announced he had terminal cancer, and was unlikely to live beyond a year.  Just two months later, he has died.

Dammit.

Read More

rainfall statistics in d3

My explorations of the d3 data visualisation library for JavaScript continue apace.  I have now added "box-and-whisker" plots to the 2012 rainfall, helping demonstrate the anomalously wet April and July, and dry August, last year.  This required calculating the various means, medians and quartiles, which wasn't particularly difficult, and has given me a feel for working with arrays.

Box and whisker plots show minimum, lower quartile, median, upper quartile, and maximum.
The mean value is overlayed as a blob.

This latest version replicates the previous chart displaying this information, which I did in Excel. (The bars are slightly different, because here I have included the 2012 data in the statistics, whereas I excluded it in the previous version.)

I think this d3 version looks much cleaner.  It certainly lets me fiddle with things at the programmatic level I prefer!

Read More

interaction with d3

My explorations of the d3 data visualisation library for JavaScript continue.  Last weekend I dipped my toe in the water, and drew some simple rainfall charts.  The code was clunky, and the data was hardwired, but the charts were pretty enough to convince me to continue.

And so this weekend I've explored some more.  I've modified the code to read in the data from a csv file, and then refactored like mad to make it more elegant.  The result of half a day's work was a web page indistinguishable (on the surface) from before: but now the charts, and the headings, are generated from the csv file data, so I can add more data each month, and the page will update without any code having to change.  So, a success.

Then I decided to add a little bit of interaction: changing the colour of a bar or spot on mouse-over, and showing the actual data value.  The d3 library is powerful enough that it was very easy to do.

in May 2007 we had 76mm of rain

in July 2009 we had 63mm of rain

So that's some tasks completed off my to-do list.  I've still got a long way to go.  But, as when I was learning Python, I'm finding StackOverflow an invaluable resource.

The actual web page is here, but it may change over time as I add more functionality.

Read More

sequestering carbon, several books at a time IV

The haul for the last fortnight.

I've already read the Saintcrow, and will be reviewing it imminently.

Read More

my first d3

Way back when I was an undergraduate, I stumbled across a book called Curve Tracing, by Percival Frost.  The college library had a first edition, published in 1872.  Several years later, I came across a newer edition in a bookshop, and grabbed at it.

This beautiful little book has tons of curves defined by equations, and shown as graphs.

Plate IV from Curve Tracing

This was all originally done in the days before computers.  It's been a goal of mine to reimplement some of this work, in an interactive form, because many of these curves have parameters that affect their detailed shape.  So, I've been looking for a suitable tool.

Also, I'm interested in data visualisation (and have long been a disciple of Tufte).  I've tried to follow the "no chartjunk" ethos in my own work.  For example, to plot our rainfall statistics, I had to struggle with Excel charts to remove most of the garish "ink" provided by default. But it's still not perfect.  Additionally, I have some specific work I want to do with modified parallel coordinates, for which there is no existing library.  So, I've been looking for a suitable tool.

Last week a colleague of mine mentioned d3, a JavaScript library for Data-Driven Documents. I browsed the gallery for a while, and fell in love.  I spent yesterday playing around with evaluating it, on simple charts, to see what it could do.  This meant reading Scott Murray's d3 tutorial, and implementing a few simple charts to show the rainfall data. (Oh, and learning my first JavaScript.)

First off, I tried a simple bar chart, starting from Murray's tutorial example (when learning a new language, I usually find it easier to modify existing code than start from an empty file). After fiddling around to get the ordinal x-axis working, I got:

d3 chart of 2012 rainfall in mm

I think that looks suitably chartjunk-free and minimal.  It also has the nice feature of combining the actual numerical values into the bars, giving what Tufte calls both a macro-reading (the bars: gosh, July was wet!) and a micro-reading (the numbers: July had 113mm of rain) in one chart.

What's nice about d3 is the way the axes scale automatically.  Exactly the same code produced these charts (the only difference is the July data value: 13, 113, 233):

automatic axis scaling with changing data values

But, of course, we have several years of rainfall data.  A little more hacking coding gave me a grouped bar chart:

rainfall, June 2005 -- May 2013

Now it's possible to see how wet April 2012 was, compared to normal Aprils, even if it wasn't as wet as that July!  My implementation of this is a little kludgy, with too much hard-wiring, since my goal was to evaluate the capability, not (yet) learn the entire language; my next task is to code it more elegantly.

I've always found grouped bar charts rather cluttered, and so I wondered if there was a better way to show the data.  Rather than use some sort of surface plot, I decide to try a projection where the size of a spot is related to the amount of rainfall.  With a surprisingly small change to the code, the grouped bar chart metamorphosed into a "blob" chart:

(left): blob area proportional to monthly rainfall; (right) blob radius proportional to monthly rainfall

This enables comparisons in both dimensions (years, or months), depending on whether you view rows or columns.

So, based on a day's work, I'm very impressed with d3.  However, there are a lot more d3 facilities I need to get up to speed with before I can start my reFrost project in earnest:

• csv data import -- currently the data is hard-wired into the scripts (ugh)
• data manipulation -- to calculate medians and quartiles for box and whisker plots
• lines -- to draw graphs rather than charts
• maths -- to calculate the functions: and, presumably I'll need a lot more JavaScript
• interaction -- so parameter values can be chosen by the user
• transitions -- so the graphs will smoothly change as parameters are varied
• more -- stuff I don't know about yet, but will need

So, a way to go, but I think I might have identified the tools I need.

Read More

all the Doctors

The Post Office has recently issued a set of Doctor Who stamps, in commemoration of the programme's 50th anniversary.

Post Office Dr Who Presentation Pack

In addition to the stamps themselves, they also issued the pictures as a set of postcards.  We got a set and had them framed.

Our Dalek seems nervous at the presence of 11 Doctors.

Read More