cue spooky music

I've written about coincidences before, and I said: "things like this happen all the time".  Just to demonstrate this a bit further, here's another one (or maybe two, or even three).

This summer we holidayed in Northumbria, and had a day on Lindisfarne.  A colleague from Scotland commented that he'd been there at exactly the same time. The final photo in the post had a couple of birds in the background; I asked a bird-savvy friend to help identify them.  When they replied, the mentioned they had been on a cycling holiday in Northumbria, and close to Lindisfarne at this time!

Last week at work we had a Christmas quiz.  One of the questions was "what element is named after a Scottish village?"  I'd never heard of this one (if they'd said Sweden instead of Scotland, I could have done better), but one of our team-mates said "Strontium".

I'd forgotten about the Lindisfarne coincidence until I received the bird friend's Christmas letter, where they mentioned their Northumbria cycling holiday. They also mentioned an earlier holiday in Scotland, where they had spent a week "in a village called Strontian".

Cue spooky music.

Read More

a jewel of a probability puzzle

I've been spending way too much time reading John Baez' Azimuth blog. It's got lots of fascinating stuff, but one piece that really caught my eye is a probability puzzle:

Suppose I have a box of jewels. The average value of a jewel in the box is \$10. I randomly pull one out of the box. What’s the probability that its value is at least \$100?

First thought, of course, is: "I don't have enough information; the answer will depend on the distribution".  But in fact you can give quite a strong bound on the answer even without knowing the distribution, using Markov's inequality: the probability is +++\le 1/10+++.

At first, this looks impossible: how can this be independent of the distribution?  Surely I can have some strange collection of jewels that violates this bound?  But no, and in the comments section there are some great explanations that give an intuition about why this is so, including David Guild's:

As long as gems have non-negative value, then (probability of pulling a \$100 or better gem > 10%) implies that (average value > \$10). Since the average is exactly \$10, then the probability can’t be more than 10%.

A very clear proof is laid out by Greg Egan. I'll rework it here, for the general case of an average value of +++a+++ (rather than the specific \$10) and a big value of +++b+++ (rather than \$100), to get the general result.

Let +++A+++ be the set of all jewels, +++\#A+++ be the size of set +++A+++, and +++V(x)+++ be the value of jewel +++x+++. Let +++B+++ be the set of "big value" jewels
$$ B = \{x\in A | V(x) \ge b\}$$We know the average value of the jewels is +++a+++:
$$ a =  \frac{1}{\#A} \sum_{x \in A} V(x) $$If we replace the sum over all jewels +++A+++ with the sum over the smaller set of jewels +++B+++, we must get a smaller answer (assuming that all the values are non-negative: necessary in the general case, and implicit in the given example).  So:
$$ a \ge  \frac{1}{\#A} \sum_{x \in B} V(x) $$From the definition of +++B+++, we have +++\forall x \in B, V(x) \ge b+++.  This implies that
$$ \frac{1}{\#A} \sum_{x \in B} V(x) \ge  \frac{1}{\#A} \sum_{x \in B} b  $$We can do this last sum:
$$ \frac{1}{\#A} \sum_{x \in B} b =  \frac{\#B}{\#A} b$$Putting this all together, we have
$$ a \ge  \frac{1}{\#A} \sum_{x \in B} V(x)  \ge \frac{1}{\#A} \sum_{x \in B} b = \frac{\#B}{\#A} b$$So
$$ a \ge  \frac{\#B}{\#A} b$$Rearranging gives the result that the proportion of big value items to all items (the probability of drawing a big value item) is:
$$ \frac{\#B}{\#A} \le \frac{a}{b}$$For the example with +++a=10+++ and +++b = 100+++, this gives us +++1/10+++.

This derivation assumes that the chance of pulling out any jewel is the same.  But, as Baez explains, the result is independent of this.  If some jewels are more likely to be picked, and that likelihood is used to define the average value too, then the result stands. (I leave the proof as an exercise for the reader.)

So, from a problem that initially looked as if it doesn't have nearly enough information, we've moved to an intuition about why a result (albeit only a bound) can be given, and a simple proof of a general result for that bound: Markov's inequality.  The power of maths!

There's also a result for values that can go negative, which requires also knowing the standard deviation: Chebyshev's  inequality.  It's all on John Baez' blog.  Go there, and you to  might spend as much time reading around as I have!

Read More

Coincidence!

I've only recently started this intermittent blog, and a few days ago I sent off an email to a friend, notifying him of its existence. We haven't communicated for a few months, so it was amusing to get back the response:

(interesting stuff about stuff ....)
Hey, I was in the middle of this when YOUR e-mail came in! Telepathy?

Now, I know he doesn't really think it's telepathy, but some people do get exercised about this sort of coincidence. But things like this happen all the time. And it's easy to see why.

It might seem like the question is: "What's the chance of me receiving an email from you just as I'm writing one to you, given we haven't emailed each other for ages?" Pretty small, I suspect. But actually, the question is really: "What's the chance of me receiving an email from you just as I'm writing one to you, given we haven't emailed each other for ages, and given that I've just received an email from you just as I'm writing you one?". That is, what is p(x|x) (the probability of x having happened, given that x has happened)? Well, it's one. You can't get less unlikely than that!

Okay, that seems a little unsatisfactory. It still seems somehow to be remarkably unlikely. What's the probability it will happen again? Very small. But what's the probability that some weird coincidence will happen again? Now that is rather high.

Let's assume that we think some event has a probability of one in a gazillion of happening (the probability prior to it actually having happened, that is). But there are equally gazillions of unlikely things that could happen. Say you get an email from a friend just as you were thinking of them. But they might have phoned you, or texted you, or visited you, or written to you. Or you might have seen them on TV, or read about them, or about someone with the same name. And you could have been thinking of any of your friends, or of anyone else, or of anything else.
There are oodles of possible unlikely coincidences. What are the odds that one of them happens?

The way probability works, it's easier to calculate the chance of none of them happening. Let's say the odds of each one of these things happening is one in N, where N is very large (one in a billion, one in a trillion, or more). So the probability is 1/N, and the probability of it not happening is 1-1/N (very nearly, but not quite, certain that it won't happen).

Now let's say the number of unlikely things that might happen is also this huge number N (it could be 10N, or N/10; the calculation is cleaner using N, but the overall flavour of the result still holds for other values).What is the probability that none of the N unlikely things happens? That is, what is the probability that the first thing doesn't happen, and the second thing doesn't happen, and ... all the way up to and the Nth thing doesn't happen?

We just multiply the individual probabilities together, so we get (1-1/N)^N. That's the probability of no coincidences, so the probability of at least one coincidence is p = 1-(1-1/N)^N. For N larger that about 100, p is about 63% (for 10N it's 99.99%, for N/10 it's 10%). That's a pretty good chance of a weird coincidence. And that's just today!

The moral is: when individual events are unlikely, but there are also a lot of events that could happen, something will almost certainly occur.

So, that email: unlikely coincidence? That coincidence, yes; some sort of coincidence happening, not really.

Read More