Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Who Reviews the Reviewers?

By DocJohn in Technology
Sat Jun 25, 2005 at 04:41:09 AM EST
Tags: Technology (all tags)
Technology

I always wondered why it seemed like every product that Infoworld's Test Center reviewed was rated either "Very Good" or "Excellent." Well, that's because 73% of the products were, with practically every other product reviewed falling into the "Good" category. What's the point of such a rating system if everything is rated the same?


This is a lesson in creating valid scientific instruments that actually measure something, rather than providing pretty numbers that purport to measure something. Infoworld is one of a number of weekly IT magazines that is distributed freely to IT professionals working in the IT world. Some cynics would argue that these types of periodicals are nothing than a set of repackaged press releases from the computer and software companies. However, Infoworld purports to be different in its Test Center reviews.

Background

Infoworld's Test Center features a set of product reviews (of computer equipment and software) in nearly every weekly issue, many times pitting similar products from different manufacturers against one another. Surprisingly, there's little information on the Infoworld website about the "Test Center," its purpose, the objectivity of its reviewers, whether Infoworld has to purchase the items it tests (ala Consumer Reports or whether they are provided free of charge to the company), etc. We did, however, find this note in the About Us section of their site under "Editorial Philosophy" (do they teach a course on that in journalism school?):

InfoWorld provides in-depth technical analysis on key products, solutions, and technologies for sound buying decisions and business gain. InfoWorld.com is the place to turn for the latest breaking news and in-depth coverage of the issues, trends, and products that run your enterprise. InfoWorld.com also features interactive discussion forums, trusted industry columnists, and incisive product test results and reviews backed by the renowned InfoWorld Test Center.


According to the Test Center scoring list, products are ranked as follows:
  • 8.7 - 10.0, Excellent
  • 8.0 - 8.6, Very Good
  • 7.0 - 7.9, Good
  • 6.0 - 6.9, Fair
  • 5.0 - 5.9, Poor
  • 0 - 4.9, Unacceptable
The first clue that something may be a little unbalanced at the Test Center is that the score ranges for each category are unequal. If this scale was derived from some sort of statistical procedure, that's understandable. But there's no information how this scale was derived, and it could've just as easily been pulled from some editorial meeting where a bunch of writers came up with it. By definition, however, we would expect that any product that is commercially viable is not going to be "unacceptable" nor likely to be tested by the Test Center. So that makes half of the scale useless as any sort of meaningful rating, since no product is likely ever to obtain a score below 5.0.

Realistically, then, all products tested will fall into one of five categories, three of which are positive (excellent, very good, or good), one neutral (fair), and one decidedly negative (poor).

Methodology

We visited Infoworld.com and clicked on the "Test Center" tab. Then we chose the "Products Review Guide" sub-tab. Going back to February 25, 2005 (the earliest reviews we could pull from its website, although it's not clear why older reviews are not available), we did a simple analysis of the results of Infoworld's tests by choosing its "Product Reviews Guide." We counted for all products listed, those that scored Excellent, Very Good, Good, Fair, Poor and Other (Beta or 0.0 scored products).

Results

We found 85 reviews listed. Of those reviews, 24 products were rated "Excellent," 38 were rated "Very Good," and 19 were rated "Good." We could only find 2 products rated "Fair" and no products that were rated "Poor" (although two were rated 0.0 and one was marked "Beta"). We also found no products rated "Unacceptable."

Discussion

From this data one can surmise that if you submit your product for review to Infoworld, you have a 73% chance of receiving either a "Very Good" or "Excellent" rating! Those are pretty good odds for any vendor. (Settle for just a "Good" rating, and your chances increase to 95%.)

Infoworld also publishes a weekly "Leaderboard" which appears to be a sampling from their Test Center reviews. In a recent issue, for instance, they listed "Hardware Servers." All six servers listed received an "Excellent" with a range of only 6 points between all products tested. In the same issue, another category listed "Hardware Printers: Monochrome" and 14 products were listed. 9 were listed as "Very Good," 1 was "Excellent," and the remaining 4 were "Good."

If we were looking at a normal bell curve distribution of this data, you would see the vast majority of products that Infoworld tests are "Very Good," with a smaller minority of products either being "Excellent" or just "Good" on either side of the distribution.

Don't get us wrong -- the narrative test reviews of the products reviewed by the Test Center are usually excellent and well-written. But it's pretty obvious their scale and scoring system needs an overhaul, as these distinctions between products have become meaningless. If the scoring system is supposed to be a short-hand for the quality and usefulness of the product (as determined by the narrative review), then it has lost its meaning. If every product you test is pretty much guaranteed to be at least "Good," then most vendors can't go wrong submitting their product for review.

We have to wonder though, if all this stuff is good, then why do people continue to have so much trouble with computer hardware and software? Is Infoworld only testing the best of the best??

But the most important question to ask of all of this is, What is the point of a rating system where it appears everyone's getting an A or B? Where are all of the C, D and F products to balance these reviews out??

This only goes to show that even established publishers can simplify things to the point of the data losing meaning.

As an aside, it's interesting to note how Infoworld blurs the line between editorial and advertising content on its website, even in the Test Center reviews table (which is an editorial table). Every entry includes a column for "Special offers," which is simply advertising tied directly to that product. Makes you wonder...

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Related Links
o Test Center
o Test Center reviews
o About Us
o Also by DocJohn


Display: Sort:
Who Reviews the Reviewers? | 56 comments (41 topical, 15 editorial, 0 hidden)
It would be interesting... (3.00 / 6) (#2)
by CanSpice on Thu Jun 23, 2005 at 05:11:01 PM EST

...if you turned this into some kind of study about reviewing. Take a look at places like IMDB, for example. If you pick a movie at random you'll probably find that its average score falls somewhere between 6 and 8. Amazon reviews are the same thing -- it's rare to find an object that's scored two or fewer stars, and most of them are going to be between three and four-and-a-half.

I noticed the same thing personally when I try to give scores to movies. I'm in a movie-watching club -- one person picks a movie on DVD and we watch and grade it. I found that I gave scores between 6 and 8 more often than not. Sure, this could be a sampling effect (people are more likely to want to show movies that they think are good, so the scores are going to be arbitrarily high), but even after I tried to recalibrate my scale I still drifted back to giving so-so movies a 6, average movies a 7, and good movies an 8.

Perhaps there's some inherent human trait to want to give ratings that are above what the mean should be on a scale. Unless something is an absolute pig, it's relatively rare to see someone actually using the whole 1-10 scale. Ideally an average movie should get a 5 and an Oscar winner a 10, but in practice that doesn't happen.

Amazon biases toward positive reviews (none / 1) (#3)
by maynard on Thu Jun 23, 2005 at 05:35:48 PM EST

One thing I've noticed about Amazon reviews is that no matter how many positive ratings (this was useful to me) a negative review gets, it will never get featured in the spotlight reviews. The system also appears to default to positive reviews first, forcing a user to manually select bad reviews in order to discover any problems with a product. I realize that amazon wants to sell good, so bad customer reviews discourage that goal. But still, I find it annoying that I have to jump through hoops to discover someone complaining about a fatal flaw with a product I'm interested in. --M

Read The Proxies, a short crime thriller.
[ Parent ]
sure (3.00 / 2) (#4)
by mpalczew on Thu Jun 23, 2005 at 05:36:26 PM EST

Ideally, one would use the whole scale, however you have to figure that when you are watching a movie you are not picking a random selection of movies.  You are going to be watching a movie you think you will like(same goes for amazon, you will buy an item you think you will like).  To give something a poor score is to admit you did a poor job in choosing an item.  So people tend to be scewed towards the 7-8 category.  The reason that really high scores are also rare is because people in general remember the memory of watching a movie and how it made them feel.  They might have just been in a very good mood at the time, and now no other movie can match it.
Also I remember one time getting a really bad Calculus teacher in college, when it was time to rate the teacher, he deserved ones all the way down the sheat, but people didn't rate him that low, because they felt sorry for him.
-- Death to all Fanatics!
[ Parent ]
True... (none / 0) (#13)
by DocJohn on Thu Jun 23, 2005 at 08:21:58 PM EST

But nobody at Infoworld (or any other computer industry rag) is going to feel sorry for some computer hardware vendor like HP or IBM, are they?

Also, does Joe Smith really care if they give a 1 to a $100 million blockbuster? No, they don't.

So one might surmise that the closer relationship one has with the subject being reviewed, the more positive the review. And I wouldn't be surprised if there was some social science research to back up that observation.

In this case, we could surmise a close relationship between the supposedly objective reviewers and the high scores consistently given to every vendor. Or, alternatively, we could surmise the validity of the scale leaves a lot to be desired.

[ Parent ]
I agree with your observation (none / 1) (#14)
by DocJohn on Thu Jun 23, 2005 at 08:25:05 PM EST

I think that amongst ordinary folks, there very much is an inherent desire to score as positively as possible when rating someone or something one has a relationship with (there's actually some research to back this up).

This doesn't explain, however, why this phenomenon would occur at a company whose job it is to offer some level of objectivity.

It's likely that the scale they're using simply isn't useful. Whether it's not calibrated correctly, or whether it has any useful meaning to its readers, it's faulty system that has no meaning. When an information instrument (a scale, in this case) loses its meaning, readers get screwed (because they continue to attribute meaning to something that has none).

[ Parent ]
Rating for an average movie (none / 1) (#42)
by LodeRunner on Fri Jun 24, 2005 at 04:16:43 PM EST

Ideally an average movie should get a 5 and an Oscar winner a 10, but in practice that doesn't happen.

I can't speak for people in general, but for me 5 is actually "below average". I know this sounds crazy, but that's a habit formed from studying in a school system that grades from 0 to 10 where you need an average of 7 to pass.

In my point of view, a grade of 5 says, roughly, that "half of the movie is bad, half is good". And that for me would be a bad movie.

Probably not very scientific, but that's how I feel. :)

---
"dude, you can't even spell your own name" -- Lode Runner
[ Parent ]

Try a different attitude (none / 0) (#51)
by rpresser on Sat Jun 25, 2005 at 10:24:40 PM EST

a "5" grade means half of all movies are better, and half are worse.

Now, as Mankind Improves, and all Bad Filmmakers are put to death in their teens, while Good Filmmakers are given huge harems and encouraged to reproduce ... movies continually get better.  But still there will be some movies better than others, and some in the middle.  The ones in the middle should STILL get 5s, even if they happen to be better than any movie  ever made before 2005.
------------
"In terms of both hyperbolic overreaching and eventual wrongness, the Permanent [Republican] Majority has set a new, and truly difficult to beat, standard." --rusty
[ Parent ]

conditioning (none / 0) (#54)
by epicedium1 on Mon Jun 27, 2005 at 08:43:11 AM EST

I think this is interesting, and in my opinion is probably true ... what I also found interesting is realising that my scale is different from yours- I think my opinions and views tend to be very strong and often polarized ... I would have no issues giving something a rating of 1 or 10, if I genuinely felt there were good reasons for it ... but when I carefully considered what I would give a film that was "enjoyable but nothing special, decent but didn't excell", it'd be 6-7, rather than 5. I guess my 5 would be "very average, not the worst film I've ever seen but I didn't particularly enjoy it", ie. a penny a dozen ... the hoards of average and forgettable films that don't quite screw up enough for me to feel they were BAD films.

[ Parent ]
We don't have that problem here at k5! (2.75 / 4) (#6)
by glor on Thu Jun 23, 2005 at 05:53:53 PM EST

Intelligent, well-written studies like this one have a 73% chance of going straight to the trash heap!

--
Disclaimer: I am not the most intelligent kuron.

Eh? (3.00 / 2) (#7)
by Pat Chalmers on Thu Jun 23, 2005 at 05:57:26 PM EST

Maybe, 95% of reviewed products receive at least a "Good" rating because...95% of reviewed products are "Good" or better?

Hardly (none / 0) (#9)
by nkyad on Thu Jun 23, 2005 at 07:38:06 PM EST

This kind of magazine is so dependent upon advertisement revenue that it is probably against their editorial policies to rate a product anything bellow Good. Besides, are you implying 95% of the products being sold to their audience are good? If so you haven't been buying much hardware lately...

Don't believe in anything you can't see, smell, touch or at the very least infer from a good particle accelerator run


[ Parent ]
Exactly. (none / 0) (#10)
by DocJohn on Thu Jun 23, 2005 at 08:14:59 PM EST

Hence my problem. Everyone within the IT industry "knows" this, yet publications such as Infoworld (and trust me, they're not the only ones who do this, just the easiest to run some data on) pretend they offer unbiased test center reviews. The fact is, management gets a hold of this stuff and doesn't know about this inherent bias. Then they hoist X product onto IT because "it got a good review in Infoworld." Yes, *every* product gets a good review in Infoworld! That's my point. This kind of lax journalism needs to be brought to light and further public scrutiny.

[ Parent ]
It is at least possible that... (none / 1) (#12)
by Pat Chalmers on Thu Jun 23, 2005 at 08:21:13 PM EST

...companies only submit decent products or pretested stuff to InfoWorld and shunt out their usual crappy shit to the consumer.

[ Parent ]
One possible explanation... (none / 0) (#15)
by DocJohn on Thu Jun 23, 2005 at 08:28:08 PM EST

And if that's true, that means the entire resource (and a good 5-10% of their weekly magazine) is useless, since it's self-selecting out the bad products or the vendors who are afraid they may have a bad product (when, in fact, they may have a really good product, but just don't know it).

In any case, it means you can't make any kind of purchasing decision based upon these reviews, similar to those you could make, for instance, after reading a far more objective Consumer Reports review on washing machines or toaster ovens.

[ Parent ]
Huh? (none / 0) (#53)
by Protagonist on Sun Jun 26, 2005 at 08:13:46 AM EST

If the explanation is that the companies send only their best products for reviewing, and that these products are indeed as good as the ratings make them out to be, how is the resource useless?

A list of great products sounds like a good resource to me.

----
Hahah! Your ferris-wheel attack is as pathetic and ineffective as your system of government!
[ Parent ]

OMGWELCOME1! (2.00 / 26) (#8)
by insomnyuk on Thu Jun 23, 2005 at 06:02:32 PM EST

Dear Sir or Madame,

Welcome to kuro5hin.org! I hope you enjoy your stay.  In a few minutes or hours, the story you have submitted will disappear from this website.  There are a few reasons for this.  Some or all of these reasons may apply to your particular case:

1. You have submitted no comments or diaries and, ex nihilo, are submitting a story. Stories contain content that is featured most prominently on this website, and this content is collectively moderated by the users of kuro5hin.org (if you are unfamiliar with the term 'moderation,' think of it as 'approval').  As a new user, you are expected to participate prior to being published.

2. The story you have submitted has links to your own website. For users and visitors of kuro5hin.org to click on these links may constitute financial gain for yourself. Thus, your 'Story' is actually an advertisement. This is colloquially referred to as spam.  You are welcome to purchase advertising space on this website if you so desire.

3. Your story is too personal or provincial to be submitted as a story to one of the appropriate sections, and would be better suited as a diary entry, because such a thing does not require user approval.

4. Your story does not meet one of any number of editorial standards. It could contain poor grammar, multiple spelling errors, or it could be too verbose, or perhaps not verbose enough. If English is your second language, you have my sympathy, but please make that clear from the beginning, and we will endeavor to help you correct any such mistakes.

5. If your story does not violate any of the above standards, it is possible that you submitted it to the wrong section. For example, a story about Hillary Clinton's prospects for the 2008 Presidential election ought to be submitted under Politics, not Science. Please re-submit your story, but this time to the proper section.

6. If your story does not violate any of the above standards, it is probably patently unoriginal, it smacks of plagiarism, or your analysis is completely lacking. Or it is possible that your story simply mirrors something from slashdot.org, fark.com, or boingboing.net, which collectively represent much of what is soulless and banal about the internet.

7. If your story does not violate any of the above standards, then it is probably going to be voted down for being a 'troll.'  This is an internet colloquialism used to describe a statement or action that at face value may appear innocuous, but has been cleverly designed to generate angry, emotional, and irrational responses, leading to arguments and disagreements usually marked by intense vitriol.  For example, an attempt at starting a debate about abortion, despite any disclaimers to the contrary, will be received as a rather unsubtle troll.

Don't worry, it will all be over soon. This story will be soon deleted and forgotten.  Better luck next time, and welcome to kuro5hin.org!

Warmest Regards,

insomnyuk

---
"There is only one honest impulse at the bottom of Puritanism, and that is the impulse to punish the man with a superior capacity for happiness." - H.L. Mencken

Oh Please! (1.50 / 2) (#20)
by Pluto on Thu Jun 23, 2005 at 11:55:11 PM EST

This critical rant, while beautifully written, is so counterproductive.

What, really, is the point of trashing a contribution in such a dismissive way?
_______________________________________
Burgeoning technologies require outlaw zones... deliberately unsupervised playgrounds for technology itself. -- William Gibson
[ Parent ]

it's a form letter for nullos (1.75 / 4) (#21)
by insomnyuk on Fri Jun 24, 2005 at 12:04:40 AM EST



---
"There is only one honest impulse at the bottom of Puritanism, and that is the impulse to punish the man with a superior capacity for happiness." - H.L. Mencken
[ Parent ]
Personnally, I hide form letters (1.50 / 2) (#32)
by lukme on Fri Jun 24, 2005 at 08:11:19 AM EST

Every form letter I have ever recieved has been pointless, just as your's is.


-----------------------------------
It's awfully hard to fly with eagles when you're a turkey.
[ Parent ]
You've surged too early (none / 1) (#48)
by toulouse on Fri Jun 24, 2005 at 08:23:54 PM EST

The problem isn't 'nullos'; it's nullo spammers, or nullos who think this place is slashdot. Just because 98% of nullo story submissions fall into the above categories, doesn't mean they all do.


--
'My god...it's full of blogs.' - ktakki
--


[ Parent ]
Rebuttal (2.66 / 3) (#24)
by LodeRunner on Fri Jun 24, 2005 at 03:49:44 AM EST

While looking at cemments/stories/diaries count is an easy way to spot spammers abusing the Edit Queue, I find the blind dismissing of any article by a "nullo" extremely annoying. The new sport in K5 is to "-1, nullo" anyone -- oh, how smart you are uncovering nullos, sir. I've even seen "-1, buy an ad" in articles which had no product references and no links to the author's webpage (wtf?).

1. You have submitted no comments or diaries - Why are users "expected to participate prior to being published"? Say someone lurks the site for a long time, but he/she's just not the kind of person who comments in forums. Then one day he/she has something relevant to say about a given subject and writes a good article, worthy of K5. Should the article be dumped just because the author happens to have "0 comments, 0 stories and 0 diaries" in his/her profile? Apparently, that's the attitude here. I strongly disagree with this.

2. The story you have submitted has links to your own website - The article in question has no such links. By the way, this item is something that has grown out of proportion. What started with "-1, buy an ad" for articles that were clearly spamvertisements for a given website, now is a hunt for any self-referential links. No care is given to check if the page linked is a harvester of Google Ads revenues or an actual relevant link which adds information to the piece. "Buy an ad" is always there, regardless.

3. Your story is too personal or provincial - The story in question is not personal.

4. Your story does not meet one of any number of editorial standards. - Does not seem to be the case.

5. If your story does not violate any of the above standards, it is possible that you submitted it to the wrong section. - The section is okay.

6. If your story does not violate any of the above standards, it is probably patently unoriginal - It's not a repost of another site, and does not seem to be plagiarism.

7. If your story does not violate any of the above standards, then it is probably going to be voted down for being a 'troll.' - Not the case either.

So, looks like you are dismissing the story purely based on "point 1", ignoring the merits of the article.

I guess it's "cool" to play the website regular, be the one who gives the rules, whatever. That's ridiculous.

---
"dude, you can't even spell your own name" -- Lode Runner
[ Parent ]

I agree with you (none / 1) (#29)
by HollyHopDrive on Fri Jun 24, 2005 at 06:48:06 AM EST

but I still had to +3 the parent post, because it was so funny.


I make too much sense to be on the Internet.
[ Parent ]

You're a disease. (none / 1) (#36)
by vera on Fri Jun 24, 2005 at 10:41:11 AM EST



[ Parent ]
Probably (none / 1) (#55)
by HollyHopDrive on Tue Jun 28, 2005 at 06:05:43 AM EST

But one with a sense of humour.


I make too much sense to be on the Internet.
[ Parent ]

next time (none / 0) (#35)
by insomnyuk on Fri Jun 24, 2005 at 09:49:30 AM EST

I will modify the form letter so that you are not offended so easily.  But then it won't be a form letter anymore.

---
"There is only one honest impulse at the bottom of Puritanism, and that is the impulse to punish the man with a superior capacity for happiness." - H.L. Mencken
[ Parent ]
discourage autoformat (nt) (none / 1) (#27)
by Phssthpok on Fri Jun 24, 2005 at 05:14:55 AM EST


____________

affective flattening has caused me to kill 11,357 people

[ Parent ]
Come on. (none / 0) (#17)
by catastrophe on Thu Jun 23, 2005 at 09:47:10 PM EST

If most of the stuff they review is good, what are they supposed to say? That it wasn't?

long-winded (2.75 / 4) (#22)
by ccdotnet on Fri Jun 24, 2005 at 12:33:07 AM EST

You've spent a lot of space telling us what we already know: a magazine (online or offline) which depends on revenue from advertising is never going to completely bucket a given product in a review.

So we don't get the truth. But we do get a rating somewhere between Good, Better, and Best - so perhaps it's still worth reading. No-one ever expects to see "we hated it".

Maybe a new product which catches the attention of a reviewer is already in a the useful/good/value category. A new product that's obviously useless/crap/over-priced might never get a review at all.

Um, okay... but... (none / 0) (#49)
by DocJohn on Fri Jun 24, 2005 at 08:56:31 PM EST

When you go to buy a new car, do you want to know that a Yugo or Suzuki is "Good" while a Ford is "Better" and Mercedes is "Best"? Is that really helpful to you??

I suspect that if such a rating system were truly helpful to most people, you have just such a system in place in more objective publications such as Consumer Reports. In the IT world, it's not always so obvious what products are "obviously useless/crap/over-priced" until a testing center, such as the one found at Infoworld, hooks them all up to their network and puts them through their paces. That saves buyers from having to buy crap only to find out it was nowhere near "Very Good," much less just plain "Good."

Sorry you viewed it as long-winded, but I felt it was important that people understand the background and purpose of the Test Center, as Infoworld tells it.

[ Parent ]
+1 SP (2.75 / 4) (#25)
by LodeRunner on Fri Jun 24, 2005 at 04:02:50 AM EST

Since the topic is ratings, here's the rationale on mine:

Interesting article. A broader analysis on reviews (covering more websites) would get a +1 FP from me. I think the current incarnation, even with a more limited scope, is good for a +1 SP. Section pages do not get enough action, so no reason to drop a well written article.

---
"dude, you can't even spell your own name" -- Lode Runner

Not surprising (3.00 / 3) (#28)
by bml on Fri Jun 24, 2005 at 05:44:39 AM EST

I suppose that if a product does what it's supossed to do without major problems, it's probably going to earn a "Good" rating. And hardware manufacturers spend a lot of time and money making sure they don't market defective products . So what's so remarkable about this?

You're mistake is expecting ratings to follow a "normal" distribution. This is not a universal pattern. Take IMDB ratings, for example. There are a lot more "bad" movies out there than there are defective hardware products, and yet ratings there peak at around 6.8. This is because a) very bad movies often don't get distributed and b) people tend to go to movies they expect to enjoy.

Sos orry but I'm voting this down. Nothing to see here.


The Internet is vast, and contains many people. This is the way of things. -- Russell Dovey

I don't know that (none / 1) (#37)
by nkyad on Fri Jun 24, 2005 at 12:11:50 PM EST

While defective hardware products rarely hit the market, any useful scale would take into account the difference among products are rate them accordingly.

For instance, a product A, that does what it is supposed to do but costs more than product B, that performs likewise, should have a lower rating. The same goes for other parameters. Also, my experience with hardware is not that smooth. Asian manufacturers, for instance, lost in a sea of self-out-sourcing (the same product will sometimes have parts and drivers from many manufacturers, some of them without reference or even a website), will produce a lot of cheap and dirty implementations of nominally correct products (for instance, a wireless card that works flawlessly, but only on the antenna line of sight and within 5 meters). Major American manufacturers will sometimes issue entire series of flawed models (an Acer notebook we had around here some years ago comes to mind).

So, I don't think it is fair to classify everything "Good" and go upwards. It only does a disservice to consumers (or readers) and puts the magazine credibility in check.

PS: as for IMDB, while far from scientific, the natural emerging scale is probably easy to map into a one to five scale (from very bad to very good).

Don't believe in anything you can't see, smell, touch or at the very least infer from a good particle accelerator run


[ Parent ]
I don't know... (none / 0) (#46)
by DocJohn on Fri Jun 24, 2005 at 07:56:02 PM EST

I don't know that I could comfortably compare a popular movie review rating system used by millions with the editorial and review policies of a print publication, developed to help educate decision makers in IT about what hardware and software products to purchase. Such decisions often involve potentially large amounts of money (more than a $10 movie ticket), and while no IT manager worth their salt is going to make it off of a single review in Infoworld alone, it certainly could be the starting point.

The fact is, rating and survey systems are designed to help more simply convey information. That is their sole purpose. When they stop conveying useful information to their intended readership, they lose their validity (and therefore, their purpose). This is true of any rating system, including IMDB's.

The argument of the article is simple -- Infoworld's rating system conveys no useful information (and probably never has), but nobody knows that. If you read the reviews every week and look at the system, you'll assume they actually do rate some products in those lower categories. You just haven't seen one this week. Then you forget about it for another week. Then another. Then, like me, you realize you've been reading this publication for years now and have never seen a truly negative review of a product.

Yes, it would be simpler if Infoworld just changed its name to "Microsoft's Infoworld." But since that's not going to happen (after all, the mask of legitimacy is more important than actual objectivity), people have to say every now again, "Hey, this emperor ain't got no clothes on!"

[ Parent ]
Couple of nits. (none / 0) (#33)
by lukme on Fri Jun 24, 2005 at 08:18:18 AM EST

Crafting valid tests for products is a difficult thing to do. Usually, different products are designed for slightly different uses (just think oven, toster oven, toster - testing how well they all heat a slice of pizza is not a great test overall).

On point that you missed was that the reviewers tend to be in bed with the advertisers. You will typically see advertisments for the products being review, and the editors may encourage slightly better reviews to get more advertisment money.



-----------------------------------
It's awfully hard to fly with eagles when you're a turkey.
You seem to overlook a basic point (3.00 / 3) (#38)
by porkchop_d_clown on Fri Jun 24, 2005 at 01:53:01 PM EST

having a limited number of pages, magazines tend not to bother printing stories about bad products even if they did play with them in the lab.

How many trolls could a true troll troll if a true troll could troll trolls?
True, but... (none / 1) (#47)
by DocJohn on Fri Jun 24, 2005 at 08:01:06 PM EST

Magazines like Infoworld are often a part of a large, international publishing house with large, complex, and rich websites.

It takes virtually nothing to post a shorter write-up of the lesser products and publish it on your website. If they did test some of these lesser products, it would be simplicity itself to publish their findings online at little added cost. Think of "Tom's Hardware Guide" or the like.

But, to me, it's not a question of them not testing poorer products, it's more a question of them not owning up to the fact that some of these products don't deserve the grades they're giving them.

[ Parent ]
Mayhaps (1.20 / 5) (#39)
by fiftee1percent on Fri Jun 24, 2005 at 02:53:32 PM EST

You get to the point here, bub. Is it the jews, liberals, or arabs backing this outfit?

_________________
I am the majority
Heritage not Hate

wow (none / 0) (#41)
by insomnyuk on Fri Jun 24, 2005 at 04:10:19 PM EST

Excuse me, but do you know this man?

--
"I'm not even here." - Antoine, Upright Citizens Brigade
[ Parent ]
Fact: (none / 1) (#50)
by Peahippo on Sat Jun 25, 2005 at 02:21:40 AM EST

Whether products are good or lousy, they are still manufactured by corporations with legal departments that are poised to sue for slander. It is impossible to objectively evaluate a corporate product therefore. Magazines like Infoworld serve no valid purpose.


i remember seeing (none / 0) (#56)
by goosedaemon on Sat Jul 02, 2005 at 11:22:35 AM EST

a magazine i used to read had a policy (they stated it somewhere) of being nice and asking the people who put out products which got a D or F or the like before publishing the review. i was going to expand on that about slander, but i think i'll just put it here instead.

[ Parent ]
They are shills, that's all. (none / 0) (#52)
by it certainly is on Sun Jun 26, 2005 at 03:29:51 AM EST

The purpose of the magazine is not to inform you about the quality of a product. Each review is a thinly veiled advertisment. It is trying to sell you that product. Therefore, all products, no matter what the body text says, must have a percentage score that inspires confidence. It's absolutely essential if you, the magazine writer, want to continue to recieve new products and advertising money.

Some magazines buck this trend, and they get all manner of shit thrown at them by the product marketeers.

kur0shin.org -- it certainly is

Godwin's law [...] is impossible to violate except with an infinitely long thread that doesn't mention nazis.

Who Reviews the Reviewers? | 56 comments (41 topical, 15 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!