I wear many hats. I’m Malt Advocate magazine’s Publisher & Editor (and owner). My wife and I own and run three WhiskyFests. I also host whisky tastings, consult for the whisky industry, and write a lot about whisky for Malt Advocate and other publications.
My most difficult job, however, is rating whiskies. Tasting whisky isn’t hard. (It’s quite fun, actually.) Writing tasting notes isn’t difficult either. Scoring whiskies isn’t hard either, because once I nose and taste a whisky, I know what I like about the whisky or don’t like about the whisky (and why).
The tough part is dealing with the aftermath when I give a whisky an inferior rating, which I often do. By inferior, I mean rating a whisky in the 70s or lower. You might think that a rating in the 70s is something that shouldn’t upset a whisky producer, but it often does. To me, rating a whisky in the 70s means, and I quote from my rating scheme definition in Malt Advocate: ”Average. No unique qualities. Flaws possible.” It’s not the end of the world.
But, many whisky companies seems to believe that all their whiskies are great, and they don’t like anyone telling them that one of their whiskies has flaws. I’ve lost a lot of advertising over the years because I gave whiskies low ratings, and it’s the main reason why most magazines that rate alcoholic beverages on a 100 point scale only publish ratings at 80 points or higher. (A leading wine magazine and a leading beer magazine immediately come to mind.)
The responses I receive from the producers are quite varied, but many lead to the same opinion: they don’t want to believe me. Here are a few examples of responses I will get.
Other people have tried it and they liked: Maybe they just didn’t have the guts to be honest with you. Or maybe they are looking for just one thing in a whisky. I know people who will like any whisky, as long as it’s really sherried. I know a guy who only likes woody whiskies, and he spends a lot of money on old whiskies.
Do a re-taste: I don’t need to do this because before I score a whisky in the 70s or lower, I automatically re-taste on a different day prior to publishing the rating, just to be certain. And if I don’t like a whisky, I always explain why I don’t like it.
Joe Schmo in some other publication gave it a good rating: There are many legitimate (and illegitimate) whisky writers rating whiskies right now, and if you look around long enough, you’ll probably find someone to say something nice about your whisky. I’m just one person, with one person’s opinion. There are many other opinions out there.
It won a medal: With competitions out there awarding medals to nearly 90% of all entrants (for a fee, or course) you should be able to win a medal somewhere. And if you don’t, I would be very concerned.
The bottom line here is that there are great whiskies, good whiskies, average whiskies, and bad whiskies (just like everything else in life). They can’t all be great. I just wish that more people in the industry would acknowledge this.
Everybody believes that their baby is the cutest baby in the entire recorded history of our species.
Reminds me of the Seinfeld episode where the couple who had the ugly baby were completely unaware of this, and everyone else thought the child’s face was repulsive. Only Kramer (naturally..) had the guile to actually exclaim in their presence that the baby was ugly.
So when you rate a whiskey anything but highly, you’ve essentially called their baby “ugly”, or at least not terribly handsome.
It’s understandable, if regrettable.
I review music for (part of) my living, and I get emails now and then from artists or fans who are either disappointed or downright nasty. After a while reviewing, I started to really keep in mind the fact that these are people who worked really really hard on their art, and spent a lot of time and thought making it a true representation of themselves.
While working on a whiskey might not require quite as much sustained work, I can only imagine what it would be like after 10 years to wait for something you’ve put thought and effort into only to find out that it is not a complete success. What do you do, start over at the drawing board and wait another ten years to hopefully get it right? So it’s no wonder that people are often disappointed with you. As long as you are reviewing their work honestly and respectfully, both by taking time with the whiskey and by presenting both the positive and negative aspects in a balanced light, you’re doing everyone a favor by helping the artform evolve.
Your annoyance begs a question: What’s the point of giving out a score? We know that just because two whiskies score a 92 doesn’t mean they’re identical, so why group them together based on some artificial barometer? Maybe I’m in the minority here, but a narrative opinion should suffice without the need for a quickie number.
Joe, Matthew: yes, it’s understandable why certain producers take a defensive posture. And I empathise with them. I get plenty of criticism regarding my magazine and festivals. No one wants to hear that someone doesn’t like what you spent a lot of time and effort on.
What makes it even more difficult for me is that many of the distillery managers, master blenders, brand managers, etc., are my friends and I know how hard they work to do the best they can.
One thing I learned from Michael Jackson and his evaluations is that there is a right way and a wrong way to say you don’t like something. He was very good at that. I try to do the same.
I’m glad you mentioned your loss of advertising because it leads into the main reason, in my opinion, all ratings are problematic. There is a reason why Consumers Union, the publishers of Consumer Reports, do not accept advertising, and purchase each product they review. It’s because they can write bad reviews and not worry about losing advertising revenue. In effect it maintains their objectivity.
I appreciate and read John’s reviews. But because Malt Advocate accepts advertising from the whisk(e)y producers they review, and because said producers provide (some) samples for free, I will never 100% trust the ratings given. I am not saying the review is dishonest or biased, I am saying that because money is involved, that makes objectivity very difficult.
I also realize that my perfect world where reviewers would be 100% financially-independent as I suggested above is also impossible. And that some compromise will probably be necessary. Lastly, please don’t take this as a personal attack or a personal insult of your objectivity. I believe that John’s reviews well researched and minimal in bias. It is the overall structure of the reviewing system that I am critiquing.
Brian, this is probably a topic for a different thread, but … I think both are needed. The narrative opinion describes how the whisky tastes. True, it also provides some degree of information on how much I like (or dislike) a whisky. But a numerical rating is more exact. And a numerical rating without narrative would be mostly useless, for the reason you just raised.
Skwang, I understand your concerns. I can only say that my goal is to keep 100% separation of “Church and State”, if you will. And like I said, it has cost me a lot of money over the years, but I have concluded that it is the only way to maintain integrity and (in the long run) be successful. That’s one of the reasons why we don’t sell label space in our reviews like so many other drinks publications do. I don’t want to blurry the line between the two.
Even drinks publications that claim their independence by not taking ad space still get money from drinks companies indirectly throught press trips, consultation fees, subscription orders, etc. The bottom line is: are you going to be honest with your evaluations regardless of whose product it is and what the financial concequences will be? I have made that vow. I can’t speak for other publications or other people.
John,
I share the same concern that some have expressed, specifically that it’s hard for any drinks writer to overcome the obvious, and, as John points out, endemic conflicts of interest. Over the years my only complaint about John’s reviews has been that they are too gentle, too generous. But we all understand that he can not bite (too hard) the hand that feeds.
John would not, could not, have developed such a loyal following if the reviews did not line up, at least roughly, with our own experience. John, I don’t think you have to defend yourself at all here. The reviews speak for themselves. I have followed John’s leads many times and have yet to be disappointed. With what we are paying for high end drinks today, John’s work is an essential filter for those of us who can not afford catch as catch can drinking at $100+ a bottle. If John were to lead me astray with any regularity, he’d lose my loyalty pretty fast.
Like the restaurant business, the theatre, the art world, bad reviews go with the territory. And yes, they can make or break a business. That’s how it works. I don’t see anything wrong with that. John clearly understands the repercussions of delivering a tough review and, as he explains below, takes extra care to make sure he’s got it right. And, like Robert Parker, I’m sure he could be persuaded to re-taste something if a reputable company or group of enthusiasts believed he had made a serious mistake.
Rock on John, and stay real. See you at the NY fest.
John,
I have always enjoyed reading the ratings section of the magazine. In fact, I think it was one of the main reasons I subscribed in the first place. The ratings are especially useful, for me at least, because I do not have an endless supply of funds to spend on whiskey. Accordingly, when the magazine reviews a bottle in the $100-200 range that I am interested in trying, I give the review a lot of weight in determining whether I will purchase the specific whiskey.
That said, I am of the opinion that the reviews would carry more weight if it was, say, the average of four different reviewer’s scores. I know that with some samples this might not be possible. However, on more widely available whiskeys, it could take into account personal preferences.
To be clear, I am not saying that personal preferences are a problem with the current system. I just know that I myself am not a fan of highly peated whiskeys, and would likely score them lower than other whiskeys.
I am interested to hear your perspective on this.
Joe, thanks for the kind words. So, the companies say I am too harsh and you say I might be a little too gentle. I don’t try to be either, but I can see both viewpoints.
Aaron, I understand your wanting to get multiple reviews on a whisky, and I encourage it. But at least for the time being, the only reviews that will appear in Malt Advocate are mine. There are only a few whisky reviewers who I respect enough to consider such an endeavor and the are either deceased (RIP Michael) or too involved with other projects. Jim Murray is one of the people I respect and I highly recommend that you buy his Whisky Bible each year.
According to wikipedia, there is a “Lake Wobegon Effect”:
The Lake Wobegon effect
The characterization of the fictional location, where “all the women are strong, all the men are good looking, and all the children are above average,” has been used to describe a real and pervasive human tendency to overestimate one’s achievements and capabilities in relation to others. The Lake Wobegon effect, where everybody claims to be above average, has been observed among drivers, CEOs, stock market analysts, college students, parents, and state education officials, among others. The effect is closely related to the Confirmation bias among others.
from: http://en.wikipedia.org/wiki/Lake_Wobegon
Maybe all these producers that think all whiskies are above average are actually based in Lake Wobegon?
~tom
http://bruichladdichblog.wordpress.com/
p.s. To answer your question seriously: Come on. Really? Not everyone likes everything. Whisky producers are surprised by this? Whisky is personal. I may like something you don’t, or hate something you liked. So what? I don’t make purchasing decisions based solely on reviews. I usually taste something that a friend has, and buy more if I like it. Or I buy some expression I’ve never had before at a bar. In the latter case, I may choose what to try based on a review, but not always.
John,
You’re absolutely right: there ARE great drams, there are good whiskies, and there are downright foul whiskies. Those of us who consider ourselves devotees of the art of whisky distilling rely on the “experts” in the business for the kinds of eloquent and descriptive tasting notes that you and others like Jim Murray and the late, great Michael Jackson provide. I’ve rarely come across a whiskey review that diverges greatly from my own opinions and I rely on the opinions of others to inform my own tastings and expand my horizons. If I didn’t do that, I’d still be stuck on Jack Daniel’s and J&B (not that there’s anything wrong with either of those brands). Distillers would do well to listen to the opinions of those who have consistently shown themselves to be accurate and fair in reviewing whiskies. They should realize that most of us trust and rely on publications like the Malt Advocate to expand our collections and our palates.
Well, I’ll get off my soapbox now.
- Josh
Tom, good point. And I agree with you. My reviews (or anyone else’s reviews) should be just one part of a whisky enthusiast’s decision-making process–if possible. I realize that sometimes you have to make a decision based on very limited information. Then, if you want to pull the trigger, you need to find a reviewer (or reviewers) whose palates are in line with yours. If that’s me, great. If it’s someone else, that’s great too!
Josh, I could’t have said it any better.
I think the problem with a numerical rating is that when you rate whisky you are rating the quality of the whisky not the deliciousness(my assumption correct me if I am wrong, but for instance if you didn’t like oranges a whisky wouldn’t get a bad score for tasting of them and a range of other flavors, but it would score badly for tasting of oranges and nothing else because it is one dimensional). I seldom seriously rate my whiskys, because it is hard work to really concentrate on what I am tasting and being objective instead of just enjoying the dram and being subjective. An over sherried or heavily peated whisky might taste delicious (especially with a cigar or chocolate or on a cold dreary day), but not rate as highly as a whisky that is more balanced. Similarly, a lighter whisky with a large cube of ice might be very refreshing on a hot day even if it isn’t very complex. A simple numerical rating doesn’t explain that the way tasting notes do, and the numerical rating is easier to focus on.
I seldom drink whisky as an end all in my activities, more often it is a part of another activity, and the whisky tasting notes do a better job of reflecting the subjective qualities of a whisky that complement the other activities that go along with the dram. I would rather see rating that were ranges like ‘must try’, highly recommended’,'recommended,’neutral’, and ‘not recommended’. I think that roughly these correspond to ranges in the 100 point scale that Malt Advocate uses, (I have never been dissatisfied with a 80 point whisky that included interesting tasting notes), but the fineness of the scale may do a disservice to people new to whisky.
Cheers, and thanks for making such a great magazine.
Hi John,
it is nothing new that whisky is all about selling the stuff and marketing it and the production side is more and more neglected.
There are developments in the Scotch whisky industry especially that are multiple reasons for concern.
Just to name a few: industrial production methods, cost cutting, loss of the art and craft of whisky making expansion beyond reason and last but first actually general loss of quality.
Macallan build capacity up to 8 million litres a year Glenlivet is blown up to double the existing capacity. You name it.
But if everything is about selling whisky people like you and the things you do gain a lot of power.
Take Ardbeg Blasda for example. Most reviews I have read are critical to some extend. Most say it is too young and to weak at 40.0% and it is too expensive.
Now if it had received only praise the malt would have been justified in everything that it really is.
Finding fault in something which is under a huge pressure to be sold to make money is deadly so the fault finder comes under attack. No matter that the faults are really there! Speaking them out and formulating them is the crime.
I have told the same to different people in the industry: if you do not want that people critisize your whiskies ther is but one way. Make better whiskies and keep your designer stuff to yourself. I could name a few brands or distilleries but I think we all know who has lowerd the standards but asks the same prices and does not want to be citisized for that.
I want to remind you to what Ulf Buxrud said about Japanese whiskies and what nonjatta reported in his blog. The Japanese go away from industrial production methods and revoke traditional ways of whisky making which have become extinct ins Scotland.
Who wins the international awards nowadays?
Hey John,
To answer your question, no, not every whisky is great, just like not every whisky is bad. People like me commend you, and are thankful for what it is that you do. So long as your remain objective and un-biased, you should have no trouble sleeping at night. Personally, I don’t think that you’re too harsh or too soft. I feel like you tell it like it is. I question anybody, no matter what their profession, who cannot handle, or even welcome a little constuctive criticism. I also feel that it’s a little arrogant of any distillery who feels that ALL of their whisky releases are great. Admitting a mistake might go a lot farther than denying it. Most people respect humbleness, not arrogance. Distilleries send you samples to review because they respect your opinion and judgement, as do all of us. If some should decide to stop sending you samples because of a poor review, that speaks more about them than it does you! Please don’t change a thing!!
Cheers,
Tony
John-
As a long time reader of Malt Advocate magazine, and now, what does John know?, I’ve always wondered where that 100 point rating will come from? From what distiller? Will it be Scotch, Bourbon or maybe Irish. Maybe in the next issue, I tell myself. I’m still waiting and hope it will be on the lower end of the price spectrum if ever it happens.
I understand the manner in which you conduct your ratings. You clearly explain this manner in the side bar of every issue in the ratings section. I understand why a $2,000 single malt Scotch and a $65 single malt Scotch can both score a 93 rating.
However, what is puzzling for me to understand is how after all your years in the business and the countless whiskies you’ve been privileged to sample, you feel a 100 point rating has not yet materialized.
In your defense, I do not believe other highly regarded whiskey writers have scored a 100 rating in their publications. Is such a standard unattainable? Are the expectations too high?
If I’ve misspoken, and something has scored a 100 would you please provide the information. I did not take the time to flip through my nearly 40 issues, but instead used my fading memory.
Highest regards, HD.
Great comments guys. With any luck, some of the whisky producers are out there lurking, reading along.
HD. I’m still searching for that perfect whisky. The recent release of Black Bowmore came close with a 97.
Hi John,
if the industry is lurking in the forums I am convinced that there are two types of industry related readers. One is the producer type the craftsman who cares for his or her product and reads comments like ours with a bleeding heart knowing too well the content of truth of comments like ours. Those readers suffer I think.
The other is the marketing type, the decision maker type who reads our critcal comments and does not give a damn. He or she has financial deadlines to meet.
One other thing springs to mind I would say.
The whiksy world is upside down. In part for the reasons mentioned above.
In years gone by there was whisky produced at distilleries like the distilleries saw fit. That whisky had to go out into the world and find its followers. Success in that enterprise was mixed some succeeded more some less so.
Some were too successful and became victims of their success.
Today there are markets taste profiles and the need to make profits at all costs.
So today whiskies are designed to meet a taste that marketing or the spirits industry thinks they must meet. The spirits industry has a keen interest to tell us what we have to like.
And then someone like you comes along and just says that a procuct that was created under great costs in which the main share is costs for marketing is nothing special.
Now we can`t have that, can we, John?
Kallaskander, yes, I agree with you that a lot of the whiskey released today is marketing-driven (i.e., the marketing people are telling the distillers what to make and how much to make).
On more than one occassion, I have had a distiller tell me that he can’t produce the amount of whisky the marketing department wants and maintain the quality of the product. One went as far as telling me (off the record, of course) to get a bottle of the first batch, because the batches after that won’t be as good.
Hi John,
sad but true. And disastrous.
It´s boom time. Whisky producers are determined to milk the market for what it`s worth. As long as new markets emerge there will be drive behind the growth of the industry. But nothing grows forever.
The racing horse is flogged on and on to go faster than the keenest gallop.
The result is forseeable.
What to do about it? Who will intervene and stop them?
It is not probable that the whisky industry will stop to take a breath all on their own.
Hi John,
if it is a quantum of solace to you let me tell you that you are not alone.
See what London based whisky exchange say about an Ardbeg named Blasda.
Ultimately the problem with Blasda is that it’s been chill-filtered AND it costs forty quid AND it’s been bottled at 40%. If you were Glenmorangie, you might be able to get away with even two of those things. But not all three at the same time. Unfortunately the overall quality/value ratio for Blasda just doesn’t stack up against the likes of Uigedail, Renaissance or Airigh nam Beist, all of which are roughly the same price. In that context, perhaps Ardbeg are victims of their own success here. The standard of recent Ardbeg releases has been extraordinarily high, and therefore new bottlings are heavily-scrutinised and judged against the lofty virtues of their predecessors.
http://blog.thewhiskyexchange.com/?p=432
I am sure that Louis Vuitton Moet Hennesy or Glenmorangie plc like to read what is written here.
But I as a consumer find it to be true and I find that I am very satisfied and grateful that somebody writes things like that.
Otherwise there will be only “great whiskies” in the future if we let the industry and their marketing get theit way.
So keep on saying as it is. If they have not enough sense to face crticism and withdraw advertisings from magazines like Maltadvocate write about that, too.
And if the industry does not want to be citicised for whiskies that are not great there is one retort. Make better whiskies.