 Foul! Unfair
Benchmarks- 3 September 1998
Dan
Knight -
- 4
September 1998. This article has generated
an unprecedented amount of email. Most of it
tells me that the iMac ships with graphics
acceleration enabled; I had read otherwise.
Another writer say the Power Mac G3 also ships
with graphics acceleration enabled, so if NSTL
tested it with graphics acceleration disabled,
they had to do that intentionally. (Or they may
have simply assumed it was disabled since they
could find no way to enable it.) Several writers
have also noted that the second page I quote
from has been revised since I quoted from it, so
some of the words I quote are no longer
present.
-
- If true,
this says very little about journalistic
integrity.
-
- On the one
hand, I'm glad to learn that Apple ships
computers with graphics acceleration on. On the
other, I wonder why the Macs didn't benchmark
better. And on the gripping hand, there's the
issue of Microsoft Office using undocumented
Windows code to optimize performance, something
they can't do on the Macintosh.
-
- And it looks
like PC Magazine may be as reliable as
ever.
I'm practically ancient for this
industry. I remember lusting after the TRS-80 in
Radio Shack flyers back in 1977. I think it was
1979 that I first put fingers to keyboard and used
a personal computer. In all the years since, computer
magazines have been my faithful companions. In the
early days, I read Byte, Creative Computing,
and Compute! When I became a DOS geek, PC
Magazine, Byte, and PC Tech Journal were
my favorites. Today I lean toward the internet for
the best and latest information, but still read
Macworld, InfoWorld, PC Magazine, and others
regularly. (And I am singularly upset with the
folks who bought Byte magazine, put it on
"hiatus,"and dropped almost every good writer they
had. When it comes back -- and it better, I'm a
paid subscriber -- it won't be the small systems
cross-platform journal I've known and loved for
almost 20 years.) Computer magazines build their
reputations on product test reports, which almost
invariably include sophisticated benchmarks that
look at the computer at both the component level
and the application level. MacBench, BYTEmark, and the ZD
suite (CPUmark, Winstone, WinMark) are designed to
fairly compare computers from different
manufacturers, with different CPUs, with different
subsystems. Best of all, BYTEmark is designed to do
that across platforms, so it can evaluate a Pentium
II v. an Alpha v. a PowerPC. That may have been the reason
for Byte's demise. BYTEmark showed the
upstart PowerPC 750 (aka G3) outperformed the
industry darling, the Pentium II. It also gave the
Alpha a real run for the money. Best of all, the
PowerPC did this with a lower clock speed, lower
power consumption, and lower chip cost. - No wonder a Win-centric
publisher bought out Byte.
But I am not here to condemn
them, but to take TechWeb and PC
Magazine to task for unfair comparisons between
the Power Mac G3 and various Pentia. John Gartner of TechWeb has been
raked over the coals by the Macintosh community not
only because he reported that testing by NSTL
(National Software Testing Laboratory) showed PCs
as faster than Macs (see his
test report), but
because a second
page of the NSTL report
clearly states, "The Mac tests were run in the
default graphics mode. Enabling graphics
acceleration mode would make full use of
high-performance adapters such as the ATI Rage II+.
. . ." (emphasis added) Of course, a lot of readers
never get to the second page to discover that this
was not a level playing field. NSTL was comparing
graphics accelerated Windows machines with Power
Macs that did not have acceleration enabled. That's
why they carefully pointed out (near the bottom of
the second page) that enabling graphics
acceleration would have changed
everything. But to add injury to insult, Mr.
Gartner followed up with Get
Real, a column in which
he said it is fair to compare computers as NSTL
did. - Sorry, John, but in all my
years of reading computer magazines, I've never
heard of anyone knowingly testing a
computer with an important feature like graphics
acceleration disabled, especially if it shipped
with the computer.
So what if it ships with
acceleration disabled? Honest testers do their best
to put every computer on a level play field, even
ones from Apple Computer. But that's not allNow it seems that PC
Magazine may have made the same mistake. In a
series of tests released on August 28
(First
looks: iMac,
iMac
v Pentium II: test
results,
iMac
v Pentium II: what the numbers
mean, iMac
v Pentium II: how we
tested), it looks like
ZD Labs also neglected to enable graphics
acceleration, instead running their tests in
"default" mode. This is the same magazine that
used to test dozens of modems or video cards side
by side, often tweaking settings to see how much
performance they could wring out. I can't believe
that today they'd rather just take equipment out of
the box and test it without a single change. I
thought we were dealing with geeks here. It's a sorry day when testers at
two leading PC labs can't be bothered to turn on
the Mac's accelerated video and judge a Macintosh
on it's true merits. Acceleration comes with the
computer -- all they have to do is turn it
on. Then again, that might make
their favorite computers look bad. Further reading- Pentium
v. PowerPC,
MacKiDo
- Motorola
and PowerPC (also
covers 680X0 series), PC Magazine
- PowerPC
v. Pentium II:
escargot?, Mac
Musings
- Great
CPUs, past and
present, John Bayko.
See especially sections on 680x0,
80x86,
PA-RISC,
Sparc,
Alpha,
PowerPC,
and Merced.
- RISC,
CISC, and Your Mac,
MacTips
- back
to Mac Musings index
-
MacPromote
Xchange-
Member of the Mac
Web Network-
- This page and site
©1998 by Dan
Knight.
|