Modern tech product reviews are flawed

Khoi Vinh on why 24 hour or even weeklong reviews are dumb:

However I’ve come to believe that there’s at least one thing wrong with this whole notion of product reviews – and with smartphone revirews in particular – and that’s that by and large they’re only ever interested in these phones when they’re brand new.

When an iPhone debuts it’s literally at the very peak of its powers. All the software that it runs has been optimized for that particular model, and as a result everything seems to run incredibly smoothly.

As time goes on though, as newer versions of the operating system roll out, as there are more and more demands put on the phone, it inevitably gets slower and less performant. A case in point: I’m upgrading to this iPhone X from a three-year old iPhone 6 Plus and for at least the last year, and especially over the last three months, it has struggled mightily to perform simple tasks like launching the camera, fetching email, even basic typing. People who have recently had the misfortune of having to use my phone tell me almost instantly, “Your phone sucks.”

You could argue that three years is an unrealistically long time to expect a smartphone to be able to keep up with the rapidly changing – and almost exponentially increasing – demands that we as users put on these devices. Personally, I would argue the opposite, that these things should be built to last at least three years, if for no other reason than as a society we shouldn’t be throwing these devices away so quickly.

This is, of course, the reason behind the odd embargo strategy Apple employed regarding the iPhone X – if you only give people an hour or at best, 24 hours, to review a device, people will still be in the honeymoon phase of owning a product, where you’re still rationalising spending €1200 for a phone (or any other high price for any other product, for that matter). Choice-supportive bias is a real thing, and each and every one of us experiences it. During this period, initial flaws aren’t as apparent, and long-term flaws or flaws that only pop up in specific situations aren’t yet taken into account. It makes the product appear better than it really is.

This is why, back when I still did reviews for OSNews, I had my own rule of using a product for at least four weeks before publishing a review. This gave me enough time to get over this initial phase, and made sure I had a more levelheaded look at the whole thing. We don’t do many reviews anymore – I have to buy everything myself, and I’m not rich – so it’s not an issue at this point, but even if companies were to approach us today for reviews, I would still ask for that four week period, and if they were to object – sorry, but no review.

This is, of course, what the major publications should’ve done. Nobody forced The Verge or whomever else to publish a review within 24 hours. The initial embargo rush is important for the bottom-line, I get that, but it still feels rather suspicious. What can you really learn about a product in just 24 hours? Can you really declare something “the best damn product Apple ever made” after using it for less than a day? At what point does writing most of the review in advance before you even receive the product in the first place, peppering it with a few paragraphs inspired by the 24 hours, cross into utter dishonesty?

By reviewing products in a day or less, popular tech media is really doing readers and consumers a huge disservice, only further strengthening the idea that the tech press is often nothing but an extension of a company’s PR department. This erodes credibility, and in turn hurts those among the media who do take their time to properly review a product.

It’s okay to not rush writing a review to meet some asinine embargo. It’s okay to not ask “how high?” when a company tells you to jump. It’s okay to publish a review a week or even a month after an embargo has been lifted. It’s okay to not post unboxing videos of non-retail boxes.

It’s okay to, sometimes, just say no.

28 Comments

  1. 2017-11-08 11:57 pm
  2. 2017-11-09 12:38 am
    • 2017-11-09 1:09 am
    • 2017-11-09 10:04 am
      • 2017-11-09 2:00 pm
    • 2017-11-10 10:03 am
  3. 2017-11-09 1:49 am
  4. 2017-11-09 4:30 am
  5. 2017-11-09 5:33 am
  6. 2017-11-09 7:06 am
  7. 2017-11-09 9:50 am
  8. 2017-11-09 11:55 am
  9. 2017-11-09 1:30 pm
  10. 2017-11-09 1:58 pm
    • 2017-11-09 4:22 pm
      • 2017-11-10 10:58 am
  11. 2017-11-09 3:32 pm
    • 2017-11-09 3:40 pm
      • 2017-11-09 4:04 pm
      • 2017-11-10 10:07 am
        • 2017-11-10 10:29 am
      • 2017-11-13 12:15 am
  12. 2017-11-09 4:19 pm
  13. 2017-11-09 6:31 pm
  14. 2017-11-09 8:11 pm
  15. 2017-11-10 10:46 am
  16. 2017-11-10 3:29 pm
  17. 2017-11-12 3:10 pm