Homebrewed Music

At first glance evaluating audio gear seems like it should be perfectly simple, all we need to do is trust our ears. But it turns out that we don’t hear with our ears, we hear with our brain and with the nervous system circuitry between our ears and brain, all in a wildly complex network of sensing, filtering, focusing, and feeding back. And this network is both amazingly sophisticated and hilariously inept at the same time, able to discriminate between small differences but also certain to occasionally make up the answers.

The Brain Is the Problem

Here’s a YouTube video featuring James D. Johnston discussing issues related to human hearing and testing. The whole video is interesting, but this link starts at the point where JJ is explaining our current understanding of human perception. He covers many points that lead directly to the need for double blind testing for audio comparisons.

If we want real answers we need to work around the known issues with human perception, issues like confirmation bias, sense memory, attention. We need to hide the labels, switch between the test item and the comparison items easily and quickly, select a particular moment for focus. And we need to do enough trials that we have some statistical confidence in the result. Meeting these requirements can be quite a challenge when comparing hardware, but when we can resolve our questions with digital recordings we can do some serious testing with a free ABX software tool.

ABX Takes Tools

ABX testing is a simple concept but a powerful one. It’s not a tool for testing preference, its only purpose is to tell if the subject can hear a difference between two samples. The subject is presented with two known samples, A and B, along with one unknown, X. The unknown X is either A or B but is unlabeled. The subject controls the switching for A to B to X, listens for as long as they wish to each sample, then identifies X as either A or B. Then a new randomly selected X is presented and the test is repeated. With digital audio files a software ABX tool is just the ticket, and it’s straightforward for the program to maintain a record of selections and compute the accuracy and confidence level for the session.

This spring REAPER forumite clepsydrae (Casey Connor) started a thread offering a free Java based ABX tool. He has enhanced the program several times and the current version is very functional for my purposes. In fact this ABX tool adds an interesting twist, a Shootout function that presents multiple source samples for comparison.

ABXer in Action

When we start the Lacinato ABX program we open on this screen:

The text gets us started with the basics of ABX. We can use a file open dialog by clicking on the A or B buttons or drag and drop our samples into the slots.

Once the files are loaded we can play them by pressing the play button for that file. There are adjustments for gain(dBFS) and time offset(s) but my usual procedure would be to carefully match levels and start positions before bringing my sample files into the program.

Notice that I’ve turned off the Hold Pos. button. I nearly always want to start each comparison at the same point, while selecting this option continues to play through the sample as we switch between candidates.

I find it very useful to identify a portion of the samples where I think I hear the most difference. The play start position slider lets me select where the sample will start playing.

This selection affects all the samples, so we can compare on a phrase or even one note easily.

Once we’ve identified an area where we hear a difference and set the slider to start there, we’re ready to begin our ABX process. When we press the Start AB*X button it changes.

Now the button offers to Play X Again and this becomes our trigger to hear sample X. I often bounce around between A, B, and X to solidify the difference in my mind.

When I have a strong feeling that I know the identity of X I press the choose button to register my choice, and that brings the Play Next X button back.

After a reasonable number of trials, say 10 to 15, I’m ready to look at my results. In this example I just made wild guesses so my results are much worse than if I’d simply flipped a coin.

The Confidence value is the key measure and should be over 95% for a solid indication that there exists a difference that is audible to the subject. Even that level of confidence is not certainty, so the more trials and more subjects the better for critical testing.

Here’s a video demonstrating a session with the Lacinato ABXer:

Here are the same source level matched files I used in this demo:

Rode NT2a
Schoeps CMC64

More Than ABX

This program is called the ABX/Shootout-er because it goes beyond the two factor ABX comparator. The author offers it as a way to test your preference objectively. He has this to say in describing the feature:

Yes, and you can also use it to do shootouts: “i think i like this last mix i did better than the others, but maybe that’s just cause it’s the last one i did…” Load up the mixes into the program, start a shootout, pick your favorite, repeat a bunch of times, and it’ll tell you which you actually like the best. (Or the least, depending on the question you want to answer.)

I used the Shootout-er function to evaluate four mics I recorded for a recent blog post.

Here are all four files loaded. I selected them all in a Windows Explorer window and dragged and dropped them on the first line of the file list. The program moved one file to each slot automatically.

Then it’s simply a matter of pressing the Start Shootout button.

Now we see the four files shuffled for our evaluation. Listen to all four then choose a preference. In some cases this might be an ideal time to use the Hold pos. or Loop features.

Each choice is registered by the program and another shuffle is executed. When a reasonable number of tests has been made or the tester burns out from the effort, hit the Show Results button to see if the preference was real or imaginary, and which sample gets the nod.

Here’s a video demo of the Lacinato Shootouter in action:

Here are the four level matched same source clips I used in the demo:

r.wav
s.wav
t.wav
u.wav

Get Real (Information)

The reason I return over and over to these blinded comparisons is simple, bad comparisons are worse than no comparison at all, they give us bad information which our brains then lock on as solid gold truth. So we actually know less after testing rather than more. In order to really hear differences in a comparison we need to get rid of the influences that can so easily override the real sound, the real information. Thanks to Casey Connor at Lacinato.com we have a terrific new tool to help us get the real information.