Whom Do You Trust: Testing the Detector Testers

A wonder of the Internet is that anyone with a computer can be an instant expert. It's up to you to figure out whom to believe. One way to check out the credibility of a source is to scrutinize their code of conduct. There are certain rules of product testing that are inviolate:

No money from the manufacturers: a cardinal rule of product testing is that the testing organization cannot solicit or accept gratuities or favors from the manufacturers whose products they are testing. Some so-called independent testers routinely solicit a fee from each participant, purportedly to defray expenses. Sorry, but when there's money changing hands, you're no longer looking at an impartial test. Even in the unlikely event that the tester somehow resists the temptation to skew the results, who'd believe them? For this reason our comparison tests are free to the manufacturers.

No manufacturers in attendance: manufacturers have their own agenda: to make their products look good. A rep watching his detector go down in flames can hardly be faulted for trying his utmost to pressure the tester into massaging the results. For this reason we never allow them to attend our comparison tests. It's asking for trouble. One Internet tester not only allows manufacturers to attend his tests, he demands it and requires payment from each for the privilege. Worse, he allows them to bring their own samples. Not surprisingly, when we test the same models, nearly all turn in far less spectacular results. The reason is no secret: given the opportunity to enter a factory hotrod, what manufacturer could resist?

We routinely conduct tests for manufacturers in private sessions, allowing them an impartial evaluation of their products, an invaluable aid in product design and marketing. This is only common sense. But when the manufacturer reps leave the test site, they take the test results with them. They paid for them and more important, we never use test results from manufacturers' sessions for our comparison tests. It wouldn't be fair.

No made-to-order product endorsements. We're accustomed to having manufacturers whose products fare well in our tests use excerpts from our stories. In the Nineties it was impossible to pick up an automobile enthusiast magazine without stumbling across words we'd written about one detector or another. Today they're equally widespread on the Internet. Not surprisingly it's always about the test winners; why would an also-ran care to remind the public that they'd lost?

We don't charge for this; after all, once our words enter the nether world of the public domain they're fair game. But others aren't so reluctant to accept cash for kind words. The tester mentioned above also offers manufacturers a package deal: pay him to attend his tests and he'll generate a seal-of-approval product endorsement suitable for advertising. Some have taken him up on the offer and witlessly display his "performance seal of approval" on their products even though our tests have shown some of their products to be spectacularly inept.

No advertising. We don't run advertisements for manufacturers, period. Magazines routinely do this and claim it has no influence on their test results but we know better. A few years ago we were summarily canned by a major magazine for which we'd conducted many detector tests over the years. The reason: a big advertiser had complained that he wasn't winning our tests and wouldn't be sending any checks until he won. And he won the very next test. His ads resumed in the next issue.

We pay for our Website through the sale of products and information. If viewers were to stop spending money, we'd rather go out of business than accept advertising dollars from manufacturers. Magazines may express no such reservations but we do.

Anonymous purchase of detectors. If we can't obtain a sample anonymously, (it's happened on rare occasions due to deadlines) we've sometimes accepted them from manufacturers. But we later purchase a second unit anonymously and test it as well. If its scores vary substantially, for better or worse, we test a third one and average the scores from the two retail samples. You'll find our backup units for sale on eBay after every test.

Some may be surprised to learn that most of our factory-supplied samples have shown pretty much the same performance as the retail pieces. That's due not so much to a heightened sense of integrity on the part of the manufacturers but more because it's getting tougher to cheat. Just about everybody has reasonable X and K-band sensitivity these days; it's not that tough. But Ka band is different. There's no cheap and simple way to rig a detector to show equal sensitivity all across that extra-wide band. So most manufacturers have to pick which part of the band to optimize: low, middle or high. They can't do all three. And we don't advertise which Ka-band frequency we'll be using.

Naturally, there have been some tricksters. For our February 2001 Automobile Magazine test one mid-priced model turned in surprising Ka-band scores, about triple the detection range we expected. So we bought a second unit and tested it. This one performed like we thought it would. When we opened the first unit's case we discovered the secret: it was a completely different platform, a three-DRO design with a larger horn, a much hotter unit than the production version. When we wrote the story, the second set of scores was used. And that was the last time we accepted a sample from this manufacturer.

Real-world test regimen. Product testing is an exacting process but the results can be tailored any way the tester desires. Want to stack the deck so that every model entered will show great detection range? Simple: make sure the test site is straight and level and, equally important, make it a short one. The worst detectors we've ever tested could hear radar in a head-to-head confrontation from at least one mile away, usually much farther. But in our difficult Around the Curve Test, that mile-plus range dropped to a few hundred feet, far too little to give the slightest protection. As a device to keep your driving record ticket-free, in this instance the detector provided no more protection than a box of Kleenex on the dash.

If in doubt, do it again. It takes us 180 man-hours to run a full test of eight units, tabulate the results and present them on our Website. If testing is suspended for any reason, the entire test must be run again, from the beginning. There's no way to stop midway, come back another day and resume it with any hope of exactly replicating the previous conditions. A radar antenna angled one degree differently will dramatically alter detection range, for example, making it impossible to compare Detector A from the first day to Detector B from the second.

One test was interrupted by a parade of tornado-chasing trucks racing past, followed minutes later by an ominous black cloud, leading us to call it quits for the day. When we reached town we were greeted by downed power lines, flattened trees and snowdrifts of golf ball-sized hail a foot deep. There was nothing to do but start the test over the next day.

The Trust Issue: Other Sources

Okay, but what about magazines that test detectors? They do of course, accept advertising dollars from the manufacturers. Are they impartial? Perhaps, but not necessarily. The world's largest auto enthusiast magazine, for example, employs a senior editor who, during much of his 30-year tenure there, has been on the payroll of one of the manufacturers. At the same time that he's been either personally conducting or heavily influencing the magazine's detector tests, he's also been writing puff pieces about his second employer, promoting their products in print and moonlighting as a technical writer and "consultant" for them. He's also conducted intelligence-gathering projects on behalf of this manufacturer, using his stature at the magazine and often assisted by the magazine's staff to gather dirt on journalists who don't find his employer's detector quite as wonderful as he does.

Perhaps it's only coincidental, but this particular manufacturer has won every high-end test the magazine has conducted during those years, a feat unmatched in the history of consumer electronics. That success may also be partly due to the way the magazine stacks the deck. When conducting comparison tests of vehicles, they adhere to rigid price guidelines; if a potential contender exceeds the upper dollar limit by a percentage point or two, it doesn't make the cut. But in their detector tests, the favored $400 model is routinely pitted against competitors costing far less--many street-priced below $150--the ethical equivalent of asking a Mazda Miata to square off against a Porsche Turbo. This same magazine was pilloried by the Wall Street Journal some years back for its staffers' incestuous relationship with other manufacturers. Although they swear they've cleaned up their act, apparently the reforms have been somewhat less than comprehensive.

Bottom line: before mindlessly accepting as gospel whatever you read, in print or on the 'Net, ask some questions about those who generated the words. Not all will stand up to scrutiny.