The Tasting Panel is one of the more divisive things we publish and in general it comes down to the Snapshot graph: do you love it/hate it/don't understand it?
So I want to take some time to delve into the graph. I'll be using the just published South African Syrah panel (above) to illustrate. Warning! Statistics and numbers ahead, proceed at your own risk.
The first thing to understand about the Snapshot is that it is designed to be obtuse. We don't believe you can reduce these wines down to one score, so we don't give you one. Instead what you see is a visual view of the group's scores.
Before diving into individual elements of each wine's output, we need to do a short primer on the statistics behind them. First, the median. The median is the midpoint of the data, so if I have the numbers 12, 15, 18, 18, 18 (the Rudi Schultz scores), the median is 18. Note that this differs from the average: medians are better at handling small data sets with outlying data. In this case, the Rudi Schultz average is 16.2, because the 12 drags the scores down. This is particularly evident when one score is way below or above the others.
The next concept to understand is percentiles. The easiest way to think about this is that the median is actually the 50th percentile, the exact middle. So if we are discussing the 75th percentile, that's the 3/4 point of the data. Back to Rudi Schultz, where you'll notice that there isn't a score higher than 18. This means that 18 is the 50th percentile and the 75th at the same time (wonderfully confusing, eh?). With five data points, it is that fourth score when they are put in order, in the case 12, 15, 18, 18, 18. If you look at the graph you can see that the next wine, the 2009 Keermont, does in fact have a different median and 75th percentile. The underlying scores here are 14, 16, 16, 17, 20.
The 25th percentile is the 1/4 point; for Rudi Schultz that is 15: 12, 15, 18, 18, 18. As with the 75th, you can have a case where the median and 25th percentile are the same. It is in fact possible, and has happened in other tasting panels, to have the 25th, median, and 75th all be the same.
The final concept to understand is outliers. These are scores that are beyond what we would expect to see given the other scores. Back to the 2009 Keermont. That 20 is actually an outlier, because the other scores are so tightly bunched around 16. This is an example of an outlier that is pretty close; the 2008 Keermont is a good example of an outlier that is obviously different from everyone else. The underlying scores here are 5, 15, 15, 17, 18.
Knowing all of this, what does the Snapshot tell you? First, look at the purple box. Here you find the median, the 25th and the 75th percentiles. The 25th percentile is the bottom (or left side) of the box, the median the dark black line, and the 75th percentile the top (or right side) of the box. The 2008 Mullineaux has all three elements. Next, you see smaller lines coming out of the box. These are the range of scores that are outside the 25th-75th percentiles. The lines going to the left are below the 25th percentile and the lines to the right are above the 75th. Note that there will be times when there are no lines at all, or just one on one side. The 2008 Mullineaux's scores are 12, 14 (25th percentile), 16 (median), 17(75th percentile), 20. The lines come out of the box to capture the 12 and 20 scores. Finally, a dot indicates an outlying score.
Using this so-called "box and whisker" graph, you can glean insights by just looking at the picture. There was a cluster of agreement about the Rudi Schultz on the high end, with some dissenters lower down (three said 18, one 15 and one 12). The 2009 Keermont was the opposite with a high outlier (one 14 and two 16s, one 17 and the 20). The 2008 Mullineaux had a wide range without much agreement. For more examples, see the explanation page on our website: http://www.sommelierjournal.com/about-us/snapshot.html.
Basically, big boxes and long whiskers means a lack of agreement while tight boxes and short whiskers means agreement. We order the wines based on the median, with the tiebreakers: 75th percentile, 25th percentile, top whisker, bottom whisker, top outlier, bottom outlier, alphabetical. Look at the first four wines again, here are the scores:
12, 15, 18, 18, 18
14, 16, 16, 17, 20
12, 14, 16, 17, 20
14, 15, 15, 19, 20
Which wine did "better?" If you just took the average, you'd say wine four and wine two tied with 16.6, then wine one at 16.2, then wine three at 15.8. I hope you can see that showing you the visual spread of the scores gives a much better sense of what the group thought. Wine four has two people who liked it a lot and that drove the average up, but three who weren't as enthusiastic. Wine two again has a minority opinion up high to drive up the average. Wine one has the largest consensus at the higher end. The picture shows you this, without having to see the scores, and is meant to illuminate the printed discussion, not replace it.
