Linked by Thom Holwerda on Wed 8th Nov 2017 23:09 UTC
In the News

Khoi Vinh on why 24 hour or even weeklong reviews are dumb:

However I've come to believe that there's at least one thing wrong with this whole notion of product reviews - and with smartphone revirews in particular - and that's that by and large they’re only ever interested in these phones when they're brand new.

When an iPhone debuts it's literally at the very peak of its powers. All the software that it runs has been optimized for that particular model, and as a result everything seems to run incredibly smoothly.

As time goes on though, as newer versions of the operating system roll out, as there are more and more demands put on the phone, it inevitably gets slower and less performant. A case in point: I'm upgrading to this iPhone X from a three-year old iPhone 6 Plus and for at least the last year, and especially over the last three months, it has struggled mightily to perform simple tasks like launching the camera, fetching email, even basic typing. People who have recently had the misfortune of having to use my phone tell me almost instantly, "Your phone sucks."

You could argue that three years is an unrealistically long time to expect a smartphone to be able to keep up with the rapidly changing - and almost exponentially increasing - demands that we as users put on these devices. Personally, I would argue the opposite, that these things should be built to last at least three years, if for no other reason than as a society we shouldn't be throwing these devices away so quickly.

This is, of course, the reason behind the odd embargo strategy Apple employed regarding the iPhone X - if you only give people an hour or at best, 24 hours, to review a device, people will still be in the honeymoon phase of owning a product, where you're still rationalising spending €1200 for a phone (or any other high price for any other product, for that matter). Choice-supportive bias is a real thing, and each and every one of us experiences it. During this period, initial flaws aren't as apparent, and long-term flaws or flaws that only pop up in specific situations aren't yet taken into account. It makes the product appear better than it really is.

This is why, back when I still did reviews for OSNews, I had my own rule of using a product for at least four weeks before publishing a review. This gave me enough time to get over this initial phase, and made sure I had a more levelheaded look at the whole thing. We don't do many reviews anymore - I have to buy everything myself, and I'm not rich - so it's not an issue at this point, but even if companies were to approach us today for reviews, I would still ask for that four week period, and if they were to object - sorry, but no review.

This is, of course, what the major publications should've done. Nobody forced The Verge or whomever else to publish a review within 24 hours. The initial embargo rush is important for the bottom-line, I get that, but it still feels rather suspicious. What can you really learn about a product in just 24 hours? Can you really declare something "the best damn product Apple ever made" after using it for less than a day? At what point does writing most of the review in advance before you even receive the product in the first place, peppering it with a few paragraphs inspired by the 24 hours, cross into utter dishonesty?

By reviewing products in a day or less, popular tech media is really doing readers and consumers a huge disservice, only further strengthening the idea that the tech press is often nothing but an extension of a company's PR department. This erodes credibility, and in turn hurts those among the media who do take their time to properly review a product.

It's okay to not rush writing a review to meet some asinine embargo. It's okay to not ask "how high?" when a company tells you to jump. It's okay to publish a review a week or even a month after an embargo has been lifted. It's okay to not post unboxing videos of non-retail boxes.

It's okay to, sometimes, just say no.

Thread beginning with comment 650749
To view parent comment, click here.
To read all comments associated with this story, please click here.
RE: Two different points
by Savior on Thu 9th Nov 2017 10:04 UTC in reply to "Two different points"
Savior
Member since:
2006-09-02

While I don't disagree with Thom's comments, they seem disconnected with what I took away from the article.

It is not just that reviewers only use the device for 24 hours. They could use it for 4 weeks and still be impressed.

The core point of the article, in my view, was that we should be writing product reviews of hardware that has already been in the market for a year or more.


I didn't read the article itself, but just from the lead quoted here, that is also what I thought the article was about. And it is a perfectly valid point, although I don't see how year-long review periods would be feasible in today's "I want it now" Zeitgeist. This might especially be true for Apple-fanbois.

Such reviews are not completely unheard of, however. At least with cars, one can find reviews that start with "I have had this car for X years...". Unfortunately, it is not the norm with other appliances.

Edited 2017-11-09 10:05 UTC

Reply Parent Score: 3

RE[2]: Two different points
by ahferroin7 on Thu 9th Nov 2017 14:00 in reply to "RE: Two different points"
ahferroin7 Member since:
2015-10-30

It really depends not just on what's being reviewed, but also on the expected product lifetime and how much the product is likely to change when it gets updated.

For most people, cars have an expected lifetime of at least five years, and potentially more than a decade. They also have a very consistent versioning cycle, and upgrades from version to version are usually small incremental changes that would be classified as practical only for a point release in most software. As a result, they tend to be given much longer periods of review prior to reviews being posted, especially because people who care about having the 'newest' car don't care as much how it will perform long term (since they're liable to just buy a new one in a year or so anyway).

You can see a similar tendency to car reviews in American reviews of firearms, which similarly to cars tend to have a long expected product lifetime and change relatively little from iteration to iteration. They tend to focus specifically on accuracy, and to a lesser degree, ergonomics, which are the two biggest factors for most people who are doing their research when trying to decide what gun to buy, and usually do the testing with a sufficient amount of ammunition (typically at least five hundred rounds, more often at least a thousand) to cover the typical wear on the gun from a owner who does a good job of properly maintaining it (which coincidentally tends to be the same type of person who does their research for something like this).

Smart phones, computers, and similar technology however, carry a general perception of a relatively short expected lifetime (often no more than six months to a year) among the people who tend to read reviews, and usually change very drastically from iteration to iteration (look for example at how much the iPod changed at each iteration, and how much the iPhone and Samsung Galaxy S phones change each year). Because of this, and the fact that most people consider 'best' to be the same as 'newest' when talking about technology (rightfully so to a certain degree), there's a much greater immediacy to the review, although I agree that the immediacy doesn't warrant only a day or two of usage.

Reply Parent Score: 2