Earlier this month I blogged about ExactChoice, a comparison shopping service for customized computer systems. I made a general comment saying that "for people with ordinary requirements a pre-packaged system may be a better deal compared to customized build-to-order systems."

Steve Krause from ExactChoice.com left me a comment yesterday:

------------------------------------------------------

Thanks for your write-up on ExactChoice. Regarding your conclusion, you might be interested in a study we did about relative price differences between packaged and customizable computers.

More info: http://www.exactchoice.com/Study1.aspx

------------------------------------------------------

My curiosity drives me to spend a couple of reading and checking the details of the referred research work. My judgment: it is not a scientific study and does not provide conclusive evidence that customized systems offer more value than packaged systems. The study falls apart because of two major issues:

First, the select sampes are not representative of the overall packaged system market.

The study is based on the price of listed systems from HP, Compaq and Sony and that of comparable customized systems. The study uses products of these three brands as the baseline because these three brands "together account for a majority of U.S. PC sales through retailers."

The problem of the study, however, is that although these three brands account for majority of US retail PC sales, one cannot conclude that they represents the entire packaged system market. (As an example, just because foreign car models take majority of the luxury car market segment, we cannot say their prices are representative of the entire segment.) It is apparent to people with some understanding of the PC industry that HP, Compaq and Sony are at the high end of the price spectrum and there are a number of named brands like Gateway, Dell and eMachines that also offer competitive value, not to mention lots of packaged systems from smaller brands.

If the study can include default models from Gateway, Dell and eMachines and alike, I am pretty sure it will not yield the same result (price is in favor of customized systems).

Second, the study makes a generic conclusion without properly statistical treatment.

In the summary the study concludes that "[t]he average price difference between packaged and customized equivalents was $110 in favor of the customized computers." However, what the study really find out is "[a]cross all 62 packaged computers, the average price difference between the packaged and customized computers was $110, in favor of the customized computers."

An in-depth look of the attached line-item details of the comparison shows that most of the price difference happens to one brand, Sony. In fact, the difference between Compaq models and comparative customized systems is only $27. $55 is for HP models and $219 for Sony models.

The numbers can at most concludes that Sony charges a hefty premium compared to the other brands. However, the study put the $110 into the summary directly, without taking time to weight the result by the relative market share of the brands. The end result is therefore skewed toward the high Sony premium. The study results favors Sony not because Sony has a larger share of the market than HP and Compaq, but simply because Sony has more models at the time of the study.

There are other problems of the study methodology, like it is only a snapshot of the market on a particular day (12/1/2003), and thus may not capture the fluctuation of the market over a period of time (pricing during the holiday season surely cannot be used to make an across-the-board claim like this).

In conclusion, I found the study is poorly designed and not providing conclusive evidence that customized systems offer more value.

This does not mean ExactChoice.com does not have value. On the contrary, I am a believer of comparison shopping and think ExactChoice does offer a valuable service to a good number of customers. The quality of the study, however, is just not up to par.

(This post is part of PFBlog Product Review series. Check out more reviews here.)