Wednesday, October 17, 2012

Do Reviews Matter?

Here is a very interesting thread over at EDCF regarding the relative merits of the Kershaw Cryo.  Some know-it-all chimes in with his big mouthed opinion (that was me, of course) and things kinda went south from there.  Kripto's point on page three with regard to my last substantive comment was well-placed (and well-deserved) and it got me thinking--do reviews matter?

The answer is no, not in the long run. But then again, depending on how you define "the long run" lots of things cease to matter.  Even in the short term, I think reviews don't really matter all that much.  Doctor saves a kid's life in the ER--that matters.  Innocent man exonerated by DNA--that matters.  Compared to those things, reviews don't matter.

But, if you are in the market to buy something, then knowledgeable and systematic assessments of things can help you out.  In that sense, reviews do matter.  If you are the kind of person that wants to make every penny go as far as it can in terms of getting you more of what you want, then why not spend a few minutes, read a review, and get another perspective?

Reviews and Objectivity

Daniel Patrick Moynihan (a Democrat that worked for two Democratic, JFK and LBJ, and two Republican Presidents, Nixon and Ford, and crossed party lines when voting) famously said: "people are entitled to their own opinions, but not their own facts." The problem is, in today's age of perpetual cable news, there are no facts of agreement anymore.  Every fact is surrounded by and contextualized by a nebula of quasi-facts and depending on where you get your facts from the cloud surrounding it takes on a different partisan feel.  If you are a liberal you get your facts from MSNBC.  If you are a conservative you get your facts from Fox News.  If you are hipster that only wants to know enough about the news to make snarky, funny comments at a party you get your facts from Jon Stewart.  This proliferation of "facts" makes it hard to determine what the heck is going on.

The Internet and forums boards have the same effect.  Everyone has an equal voice and those willing to type the most seem to be the loudest.  But the truth is, in this morass of opinion, there are still undeniable facts.  These facts should form the basis of any review and I do my very best to tailor what I write to represent those facts.

Reviews cannot be objective.  If they were, they would read like spec sheets.  "Light A produces 300 lumens on high, 100 on medium and 1 on low." Reviews, at their heart, require analysis and analysis requires both facts and opinion.  Ours is an age of relativity, in part because of the lack of facts of agreement, but also because anyone can get access to a soap box.  I did.  I paid $10 got a domain name and started this blog.  I have no expertise.  I have no engineering background.  I do try to write clearly, have an honest review process, and ground my opinions in fact, but other than that there is no secret.  I don't have special knowledge and I have very limited special access (it is true that companies send me products to review and BladeHQ is kind enough to do the same), but my opinion is no better than anyone else's opinion.

What I hope to do is to set my opinion apart by writing well, being transparent, and being systematic.   I hope that by hewing as close to the relevant facts as possible, you can understand my opinion, and hopefully agree with it.  All reviews are a form of persuasive writing and in writing as in film, it is always better to show and not tell.  I could tell you something is a piece of crap, or I could show you all of the things that led me to that conclusion and you yourself could form that opinion on your own.  I hope I do the latter, because, after all, it is vastly more persuasive to lay things out and have you put them together in the same way.  It is like a mini science experiment.  Repeatability demonstrates reliability.      

This, of course, begs another question--whose reviews matters?  For me, there is a sensible answer.  Three things count for me when I am trying to figure out whose opinion to listen to:

1.  Does the person have a lot of relevant experience?
2.  Does the person have tastes and objectives similar to mine?
3.  Does the person arrive at their opinions in a fair, systematic, and transparent way?

The closer the person is to matching all three of those things the more likely I am to value their opinion.  NOTE: all of the reviewers in this article can be found on the links page of this site. 

Experience

If you could, wouldn't you want to test out every single product in the class of products your looking at and decide based on that which one you want to buy?  Unfortunately, none of us can test every knife or light out there for a month, decide, based on that, and buy what we want.  There is not enough time nor do most of us have enough cash.  Instead, reviews and reviewers allow us to short circuit the product deluge and present us with Cliff's Notes versions of those month-long trial periods.
Better reviewers have a wide range of experiences with products.  The more experience the better.  A guy like Selfbuilt is a perfect example of this.  Selfbuilt's reputation among the flashaholics around the world is second to none.  Companies send him review copies of virtually every light being produced and the end result is a MASSIVE library of reviews.  If there is a production light released, chances are Selfbuilt will get his hands on it.

Over time that experience creates a certain basis of knowledge.  You learn what works and what doesn't, what's important and what's just marketing hype.  While I have no where near the library of reviews Selfbuilt does, I do see this refinement-of-knowledge process happening in my own reviews.  After 30 folding knife reviews, I have a pretty good sense of how steels perform.  Now sometimes performance varies, like I have seen with 154CM, and that is something that is challenging for reviews.  But if you are consumer and you don't have access to lots and lots of review samples, that challenge can be virtually insurmountable.  When I bought and used by Sequel, with its lemon batch of 154 CM, I had such a negative experience that it turned me off to the steel and Benchmade for more than two years.  Having used more knives with this steel (the blade on the Skeletool CX and the Mini Grip) has shown me that the experience was just an aberration, but without more data points, one random bad experience taints a person's assessment of things.

The more experience the better because with that experience comes a certain kind of knowledge, maybe not expertise exactly, but something like more representative knowledge.  This more representative knowledge is based on an increased sample size and with a larger sample size the knowledge one has becomes more accurate and less likely to be polluted by random good or bad.

Roger Ebert has watched probably something like 20,000 movies.  Does that, alone, make him a good reviewer?  Nope.  But 20,000 movies can't make him worse.  In a reviewer, the more experience the better.   

Similar Tastes

Nutnfancy is probably the most important single reviewer of gear and EDC stuff.  He has a massive audience, great access to stuff, a systematic approach, and a pretty transparent set of tests and standards.  My big issue with him is that I don't think he places enough emphasis on quality.  For him there seems to be a threshold and once met, nothing else, performance or quality-wise matters.  I understand that approach.  It is very helpful.  I still religiously watch his reviews.  But for me, he has less sway because his tastes and objectives diverge pretty significantly from my own.

Compare Nutnfancy then to a reviewer like Oromoto.  Oromoto's channel is populated with exotic and flashy custom blades.  Its rare for a knife to be in front of his camera with a price tag less than a kilobuck.  I can appreciate his tastes.  He certainly places an emphasis on quality and uniqueness, but again, I find something lacking.  Sometimes the best knife is the one that has been constantly and thorough vetted for 20 or so years, regardless of whether it has super conductor bolsters.  It may not cause heart palpitations in bling fans, but remember, knives are, at root designed to cut stuff.

Both Nutnfancy and Oromoto are excellent reviewers but their tastes vary a bit from mine.  My tastes synch up more with a reviewer like Spydercollector.  Its not just his love of Spyderco knives that does it but is appreciation of small, well built knives and knives that are strictly designed for utility tasks.  He also a clean aesthetic sensibility in his photography, presentation, and preferences.       

Fair, Systematic, and Transparent

This is perhaps the most difficult thing to figure out about a reviewer.  What do they do to test their review samples?  How do they get them?  What do they do with them afterwards?  How do they assess them prior to the review.

I can't speak to others, but with me it is pretty simple.  Most of my review samples either come from the company that made them, BladeHQ, or I purchased them or got them in trade.  I always note in the review where they come from so that you know.  I also have a strict policy of not benefiting in any way from the site, let alone from the reviews.  I give away every single item that I have been allowed to keep and that tradition will continue.  Furthermore, I don't benefit from the AdSense money or the BladeHQ commissions.  Those funds all go towards other giveaways.  Right now I have given away about $600 worth of gear and I hope to keep adding to that number.  In terms of tests, I always discuss what I did in a given review, what I cut, how I carried and used the item, how it performed in various tasks.  I do not write reviews without having experience with the product, usually about two weeks of exclusive carry.  In terms of assessment, I created the scoring system so that the method used in a review is clear.  You want to figure out what I liked and didn't like, you can read the review itself or just check the scores in each category.

My hope is that the reviewing system I use is as clear, fair, systematic and transparent as possible.  I hope that you never have a suspicion that something got an unfairly high score because of its source or its brand.

Conclusion

All of this is a long winded way of saying that reviews matter, to the extent that you care about how you spend your money.  They also matter to the extent that they are done correctly, match your tastes, and are based on relevant experience.

The Cryo thread is a perfect case in point about how to process reviews.  Lots of people aren't as finnicky as I am about fit and finish.  I get that.  A lot of people aren't willing to pay $300 for a knife no matter how good it is.  I also get that.  I understand, as well, that a lot of this is preference.  Some folks like heavy knives, some don't.

But in this morass of opinions, there are facts.  The fact is that the Cryo is very heavy for its size.  I also believe that few people want a heavy knife.  I think, in fact, they want a sturdy knife and that usually requires a good deal of bulk, so for these folks heavy isn't an issue, its a side effect of sturdy.  But if you went to these folks and said you can have a sturdy knife that weighs 3.0 ounces or one that weighs 4.5 ounces, all other things being equal, no one would choose the heavy knife.  The facts are also unavoidable when it comes to the awful thumb studs.  They do not work as blade stops, so that is not a reason to have them.  They snag on the pocket because of their size, making them less than optimal.  Finally, they are just inferior as a deployment method on the knife when compared to the flipper.  The thumb studs stink.  That last part is an opinion, but all of the supporting elements for that conclusion are facts.

Disagreeing over the quality of gear is part of the appeal of tools and finding what works for you.  But objective assessment also has a part.  This is not all opinion.  Realizing that can help you save money and get more of what you want.  I don't think people are dumb because they like the Cryo.  I think Kershaw could have done better.  That's it.  No personal judgments or slights.

In the end reviews help.  It is always nice to get some insights from people with experience and knowledge, whether it about buying a car or purchasing a flashlight.     

11 comments:

  1. Well said, as always. I wish you'd get rid of the black background, though. It's hell on the eyes when you close the window - because inevitably the next thing I see is blinding white.

    ReplyDelete
    Replies
    1. I used to have the background as a light gray and the text in black and that was roundly disliked. I'll take a peek at changing things. Any color preferences?

      Also, the mobile version of the site is white on black instead of black on white.

      Delete
    2. As long as we're talking about the site, Tony, the "video reviews" link at the top isn't working for me and hasn't been for a while. On that note, please make more video reviews!

      That's actually an interesting distinction for me. I read reviews of gear I've never owned and probably will never own. Your reviews specifically have inspired a few purchases for me (the quantum DD and the cadet, which I had been eyeing for a while but on the fence about).

      I think video reviews fall into a different category. Do you follow Going Gear's youtube channel? They just switched it up but for a long time the reviews were a lot like the "spec sheet" objective reviews you described. Now that comes I think from the fact that Marshall is selling everything he reviews on the youtube page so he's obviously not gonna pan any of his own products, but I still enjoy those reviews for one major reason, and I think it's the reason people watch Nutnfancy's feature length reviews: it's a chance to see the product in action. Regardless of what the person's saying about them. I love watching someone operate the light's UI in front of me, or flip the knife open, because we're involved in a hobby that is 50% online and I don't get a chance to handle gear in person before I buy it. For the most part, that is, we can all pick up a swiss army knife at wal-mart, but I think very, very few people have the chance to hear a knife click open or feel how responsive a flashlight's clicky is in person before they buy it. That's why I read reviews and especially why I watch them.

      Didn't mean for this to become a novel. It's my long-winded way of saying thanks and keep it up!

      Delete
    3. Video link fixed.

      I am working on a few more video reviews, specifically one of the TuffThumbz Vantage. I like to do ones of rarer or unusual items, but I guess I could start reviewing other stuff as well. It is hard to find 10 minutes of silence with a two year old in the house.

      Delete
    4. I agree with "unknown" that white on black is easy to read, but since I get to these reviews from my rss feed - when I close the window Im blinded with massive white background of my rss feeds again. So, its really more my problem than yours. :)

      I suppose I could override the style myself, in my browser. I like the style it is now, its just hell on the eyes when I close the window. :)

      Delete
  2. Black on white is great! That color scheme is proven to provide the best contrast and least eyestrain for high-volume text on modern LCD screens of all sizes. Tab-closing-flashbulb effect notwithstanding, of course.

    ReplyDelete
  3. agree black and white is ok, but Tony your replys in the white box with light grey is difficult, I need to highlight to read.

    As far as reviews go, I enjoy yours. We share a similar taste in equipment. I read your review on the cryo and then went and handled one at the store. had to agree with the weight. It really was a chunk. Also not a big fan of the assist. Maybe it was the weight or assist or something that put me off but it just did not have the feel as a byrd meadowlark G-10 that I carry sometimes. that is my benchmark for a good $25 knife.

    ReplyDelete
  4. I enjoy the reviews, Tony. I do dislike the white/grey reply boxes because they're hard to read. Why not just go for black type in those boxes? Simple opposition against the normal black background, a negative version of the site's basic style would be utilitarian and still make your replies distinctive.

    Now you have gone and made weigh some knives, so I have to go describe that somewhere appropriate.

    ReplyDelete
    Replies
    1. I'd really like to change that too and I have sent Blogger a few emails about it, but I can't seem to change it myself and when I try to screw around with the HTML it goes haywire.

      Delete
  5. No idea how blogger works but you might be able to just override the styles that are used on the page.

    Your style sheet is included in full inline in the page, so try and see if can override the style as below; or inject the following before the page starts; that'll override the background colour

    <style>
    .comments .comment-thread.inline-thread {
    background-color: inherit !important;
    }
    </style>

    ReplyDelete
  6. I have a huge respect for your reviews. I really appreciate your ability to be objective and still have a "point of view". A difficult balance to achieve but one that you do really well because of how clear you are about your "point of view" which means it never gets in the way.

    ReplyDelete