On a recent episode of the SoundStage! Audiophile Podcast, I mentioned to Brent Butterworth that we need an Audiophile Baloney Detection Toolkit. If you’re not familiar with the reference, it comes from “The Fine Art of Baloney Detection,” an essay by Carl Sagan from his final book, The Demon-Haunted World.
After that passing reference—uttered almost as a joke—I couldn’t get the thought out of my mind, so while recovering from a recent surgery, I pulled my well-worn copy of the book off my bedroom bookshelf with the goal of sketching out some modified versions of Sagan’s guiding principles. I figured it wouldn’t be difficult to take his guidelines for critical thought and skepticism and adapt them for the world of hi-fi.
As I flipped through the essay, though, I quickly realized something. I’ve spent the last 25 years or so reflecting on Sagan’s advice in the context of climatology, medicine, cosmology, foreign and domestic policy, and so forth. But looking at it from a different angle made me appreciate just how broadly applicable this “toolkit” is to pretty much any area of interest where there’s profit to be made in deception—which is pretty much all of them.
So instead of adapting Sagan’s Baloney Detection Toolkit, I thought I would present his words as originally written, although not necessarily in their original order, followed by intermittent commentary from me about how they apply to audio. Keep in mind—these are tools, not edicts or commandments. The goal here is for us to learn how to think about controversial claims, not what to think about them.
And we’ll start with one of the most obvious:
- Wherever possible there must be independent confirmation of the “facts.”
To this I would also add “whenever relevant.” Because the reality is that we don’t necessarily need extensive reports from multiple parties to confirm that a class-AB amp that’s nothing more than a variant of last year’s design can get its job done.
But if someone comes out with a new doodad that claims to do something substantially different from other doodads, backed up by no published measurements or blind listening tests, you should be suspicious.
- Encourage substantive debate on the evidence by knowledgeable proponents of all points of view.
Of all the tools in Sagan’s toolkit, this one has been most misused by ne’er-do-wells looking to muddy the waters. What these people ignore is the “knowledgeable proponents” bit. A debate between someone who knows what they’re talking about and someone who doesn’t may be entertaining, but it’s not informative.
When room correction was still relatively nascent and just starting to get good, and I wanted to learn more about it, I turned to three friends in the industry whom I knew had different takes on the subject: Anthony Grimani of MSR Acoustics, Paul Hales of Pro Audio Technology / Theory Audio Design, and Jon Herron of Trinnov. Had I the resources and the platform, I’d love to get those three on a stage together to discuss the pros and cons of frequency- and time-domain DSP, finite and infinite impulse-response filters, etc. I already know which of the three I agree with the most, but the fact of the matter is that learning from all three of them informed me in invaluable ways, mostly because they don’t all agree, but they know their stuff.
Interestingly, though, that leads right into the next item in Sagan’s toolkit:
- Arguments from authority carry little weight—“authorities” have made mistakes in the past. They will do so again in the future.
“Perhaps a better way to say it is that in science there are no authorities,” Sagan continues. “At most, there are experts.”
In other words, “Tony Grimani recommends . . .” is probably pretty damned rock-solid advice. “Tony Grimani has demonstrated . . .” is guidance you can almost certainly take to the bank when it comes to room acoustics. But “Tony Grimani says . . .” is not an argument, and I think he’ll be the first to agree with me on that.
- Quantify. If whatever it is you’re explaining has some measure, some numerical quantity attached to it, you’ll be much better able to discriminate among competing hypotheses.
“What is vague and qualitative is open to many explanations,” Sagan continues. “Of course there are truths to be sought in the many qualitative issues we are obliged to confront, but finding them is more challenging.”
This advice in particular is the subject of all manner of strawman arguments from gurus who want to be perceived as keepers of esoteric knowledge that you simply aren’t savvy enough to suss out on your own. In a recent blog post on The Absolute Sound’s website, Tom Martin—head of the philosophy department, apparently—describes “a kind of cognitive bias that favors quantification over other forms of information. Numbers seem so clear and pristine that I think this bias is understandable. But it can easily lead us astray. I want to show an example to help explain this.” He goes on to present really bad measurements as an argument for why measurements are bad, all while claiming that he’s, like, totally pro-measurement, bruh.
Look, there are any number of reasons why a publication might not do measurements. They may not have the resources. They may not have the expertise. (Goodness knows I don’t. I couldn’t do the measurements that Diego Estan does to accompany my reviews—not without his help and not without his lab equipment.) They may just not be interested. That’s all totally fine.
But when you come across someone who’s actively hostile to quantification, it’s not unreasonable for you to assume that this person either doesn’t want their subjective impressions undermined by objective measurements (I’ve been there; it’s embarrassing) or they’re selling you something they don’t believe in themselves.
Not saying those are the only possible explanations; I’m merely saying they’re not unreasonable assumptions.
- Spin more than one hypothesis. If there’s something to be explained, think of all the different ways in which it could be explained.
As a result of the recent MoFi kerfuffle, there’s been a lot of discussion about who can hear the artifacts created by digitization and who can’t.
Let’s say you’re someone who believes in your own ability to hear the difference between, say, a reel-to-reel tape and a high-quality digital copy of that tape. Let’s say you’ve done your own tests at home and proven to your own satisfaction that you can hear the difference.
Of all the reasons this may be true, “digital = ooky” is merely one. It could also be that you’re using an NOS DAC that introduces its own colorations to the signal chain, or that your perfectly modern DAC has a user-selectable filter that, say, starts rolling off high frequencies at 10kHz. I’ve seen ’em. I’ve reviewed ’em. They exist in the 21st century.
“Then think of tests by which you might systematically disprove each of the alternatives,” Sagan continues. “What survives, the hypothesis that resists disproof in this Darwinian selection among ‘multiple working hypotheses,’ has a much better chance of being the right answer than if you had simply run with the first idea that caught your fancy.”
- Try not to get overly attached to a hypothesis just because it’s yours. It’s only a way station in the pursuit of knowledge. Ask yourself why you like the idea. Compare it fairly with the alternatives. See if you can find reasons for rejecting it. If you don’t, others will.
With this one, I’m reminded of an old Upton Sinclair quote: “It is difficult to get a man to understand something when his salary depends on his not understanding it.” The truth is, a lot of people who believe specific things about audio aren’t necessarily motivated by their paycheck, but they have formed their ideas based on the teachings of those who are.
Sometimes, though, people with a financial interest in one particular position nonetheless change their position. I’m thinking of Mark Waldrep in particular. Two decades ago, Mark was one of the first people to really sell me hard on the concept of higher-than-CD-quality sample rates and word lengths at a time when I was, at best, agnostic on the issue. His AIX DVD-Audio/Video discs were, to my mind, a pretty convincing argument for his position. Now, though, I’d say Mark is one of the biggest proponents of the position that, although high-res is valuable in the recording and production of music, there’s not a lot of demonstrable value in high-res playback for most listeners on most systems.
Forget whether or not you agree with Mark. That’s not the point I’m making. What I’m saying is that he had a position and a financial interest in that position, and he changed his position anyway.
So if you firmly believe that MQA is truly a better representation of an original master recording than a lossless file, or even a well-encoded VBR AAC file, OK. I’m totally willing to have a conversation with you about that. But if you’re not willing to entertain the notion that you might be wrong, we almost certainly have nothing to talk about.
Which leads me to:
- Always ask whether the hypothesis can be, at least in principle, falsified.
“Propositions that are untestable, unfalsifiable, are not worth much,” Sagan writes. “Consider the grand idea that our universe and everything in it is just an elementary particle—an electron, say—in a much bigger cosmos. But if we can never acquire information from outside our universe, is not the idea incapable of disproof?”
The problem here isn’t so much with people who make claims that cannot be tested, but those who deny that anything can be tested to their satisfaction. I recently got into an online argument (on YouTube, maybe?) with a gentleman who claimed that if you swapped his five-nines (99.999 percent pure) silver speaker cables with mere four-nines (merely 99.99 percent pure) silver, he would hear the difference immediately.
When jokingly asked if he’d be willing to submit to a blind listening test to corroborate those claims, he could have blustered and said, “Hell yes I will!” It’s the internet, after all. He wasn’t using his real name. What were the rest of us gonna do: send the hi-fi police after him to force him to submit? Instead, though, he began a full-throated assault on the veracity of double-blind testing.
This is a person so unwilling to entertain the notion that he might be wrong that he’s ready to throw logic and reason straight out the window in defense of his position.
And here’s the thing: I have to accept the possibility, however slight, that perhaps he’s right. Maybe he can tell five-nines silver from four-nines silver just by listening to the sound of his speakers for a few moments. But if he’s unwilling for that assertion to be falsified, or to even admit that it could be falsified, it’s really not worth taking too seriously.
. . . Dennis Burger
dennisb@soundstagenetwork.com