Home |
Search |
Today's Posts |
#1
|
|||
|
|||
Comments about Blind Testing
First I can say that I don't have much of a bias for or against any
type of speaker wire (or speaker cable system with terminations) or any low voltage interconnect cable except I'd like the former to deliver as much current as possible without changing impedence and the latter to be super well shielded and matched to the components being connected in terms the capacitative and inductive characteristics the component's designers considered appropriate when the preamp, tuner or other was designed and when it's specifications were determined. I have been given dozens of different kinds of speaker wire and cable systems to use in my CES booth displays and to be used when I did speaker selling demos in stores. 20 years of research testing loudspeakers for seller and buyers showed me that the distortions in most speakers are so huge that it becomes nearly impossible to hear any unique characteristics of speaker wire except its ability to deliver current to cone speakers and voltage to electrostatics. The very few loudspeakers that might be able to demonstrate whether there was any audible improvement using one kind or brand of wire (or even a cable system including unique terminations), like the Quad 63 have other electrical components like transformers in their systems making it seems unlikely that subtle differences in wire can be significant enough to hear. I believe this to be fact because of testing that I've done when designing loudspeaker systems for audio companies, testing done as the loudspeaker system development Imagineer for Disney during the building of EPCOT and Tokyo Disneyland, and testing I've done for the AES paper I delivered March 4, 1982 in Montreaux and later that April to the Los Angeles chapter that showed that voice coils in loudspeakers undergo so many changes of "sound reproduction capability", that hearing the difference created by different wires of the same guage and current carrying capacity would be like finding the molecules of a specific raindrop after it had fallen into the ocean. When I gave my paper on loudspeaker compression to the AES chapter in Los Angeles in April 1982 a few of the engineers in attendance calculated that the just the factors of non-constant dynamic impedence and compression changes in loudspeakers that my testing showed, would likely show up as fast variations in temperature on the voice coil in excess of 100 degrees centigrade. Almost immediately the chief engineer of Cetec-Gauss stood up to mention that while he didn't think that it had really been important to mention his company's testing in this arena up to that point in time, he could verfiy that fast changes in voice coil temperatures of well in excess of 100 degrees centigrade had been measured in many of the tests Cetec-Gauss had done on their own products and those of their competitors. These kinds of massive forces obliterate "subtle" differences. The louspeakers being discussed at that moment were large voice coil types with designs that made cooling one of the most important considerations (although I've tested hundreds of different component loudspeakers for professional and home hifi use). The loudspeakers used in home hifi systems do not cool nearly as well as pro loudspeakers, and the negative effects on sonic characteristics due to temperature change are much greater than those demonstrated by professional loudspeakers. And that is just one of the factors involved with loudspeaker compression. These changes in sonic characteristics make it nearly impossible to do any real research on speaker wires that could be relevant to audiophile listening because the test to make such comparison "fair" would be literally impossible to design. By the time listeners could focus on the sound playback of of one of the test wire/products the second product in the test would already be unfairly tested because the test loudspeaker system (acoustic microscope equivalent) will not likely "sound" the same as it did 30 seconds ago. This means that the test passage would need to be made longer and restarted after a specific cooling off time and by then the human acoustic memory is gone. This could be why so many anecdotal testimonials involve hearing "things" after the time was taken to disconnect one set of speaker wires and connect a second set. The speakers probably cooled down and sounded better after the "wire changing period". Maybe comparitive testing of large voltage wire should be done using thin film or electrostatic headphones because these devices are actually better at exposing tiny differences in audio characteristics. As an aside I might suggest that the way some speaker cable systems are produced they seem to want to be part of the crossover for the louspeaker or maybe adjust the frequency response. I've always wanted speaker wire to be able to deliver gobs of current to some of the loudspeakers I've designed or listened to, except with electrostatics when I want the speaker wire to conduct the correct voltage from the amp to the speaker's input irrespective of the changes in the input impedence of the speaker. If there really was a wire that was substantially better sonically then "super consumers" like Disney would do the testing (at whatever cost) so that they could deliver the AES papers that would make greater prestige for Disney. Disney has bought numerous "home hifi" products and tested them (and sometimes even used them) because they have an incredible investment in not only having the best of "whatever" in their facilities, but in knowing more than any other audio consumer in the world. Matsu****a may be an incredible audio manufacturer and their leaf tweeter with extended response to 80kHz is one of the few devices that can be used to test whether human hearing really processes 20+kHz information when our brain decides on how much "acoustic reality" is being reproduced. But even Matsu****a has recognized that no company is as concerned with knowing what is what in audio as much as Disney. And cost is no object for Disney, so whether speaker wire was $.50 a foot or $5.00 a foot wouldn't matter to a Disney engineer because (through AES) Disney promotes their use of the audio equipment and materials proven to be sonically superior in their theme parks. The test labs at WED Imagineering are better outfitted by Bruel and Kjaer than almost any audio manufacturer in the world. As "super-consumers", Disney is unrivalled and there isn't really much difference in speaker wire sonically or there would be Disney engineers giving papers on their "wire" findings at AES conventions. As it is, Disney has strict rules about having enough copper to carry the current or voltage to the speakers designed into any facility or show in their theme parks, but even in the most critical applications there is no special type of wire specified. It's also a sure bet that the manufacturers of wire or wiring harnesses for speakers don't really want there to ever be any one special kind of wire or wiring system that proves to be superior to others. In the audiophile marketplace, confusion about which wire is best increases sales. Otherwise it wouldn't be possible to sell the same consumers 3 or 4 different sets of wire or interconnect cable at enormous cost. And the different types of manufacturers all have their own reasons for not wanting any one type of wire to be determined "best". These reasons vary even if sometimes when confronted by an insecure retailer, a manufacturer may SAY their wire is superior or the most cost effective sonically, but like I say, "Confusion increases sales" so doing a real comparison test is the opposite of what these manufacturers want. Higher sales keep companies alive and that is their prime directive, so confusion in the marketplace best fulfills that directive. Look at all the possibilities. The wire seller who truly believes his product is superior won't want to bother wasting time, money and effort to prove something he already believes. The wire seller who knows his product is NOT superior won't want to be shown up. The wire seller who doesn't care if his product is superior won't want to bother with something as unimportant as testing, especially because of the costs involved and the possibility he MAY be shown up. So it is against the interests of wire sellers to participate in any tests which would clearly determine which speaker wire was superior or at least which was best for which speaker or for the money. Even if one crusading wire maker were to support a true test of wire it would only allow every other manufacturer to look at the results and then make a copy of the product. And that doesn't consider the great unknown that could be very deadly to any and all of the wire companies. It can happen in the most bizarre ways but the results can be quite staggering. In 1978 I was the professional products marketing manager at ESS (Electrostatic Sound Systems) at a time when they sold professional versions of the Heil tweeter, some professional amplifiers and had the rights to import some European professional products. The company took on a new ad agency for consumer products and this agency did a month of field research in 20 retail outlets to see what they had to work with for ESS' new ad/marketing campaign. The ad agency found that in head to head sales demos using ESS speakers against any other speaker brand of comparable price, store salespeople were able to sell the sound superiority of ESS loudspeakers against any other brand. With the various retailers around the country carrying various mixes of loudspeaker lines, ESS seemed to be able to sell their products against any other brand based on sound quality during a demo. Store salesmen usually take the path of least resistance and they would have sold more ESS loudspeakers except for one thing. Sometimes when store salespeople suggested to potential customers that they listen to speakers, the customers didn't want to listen to ESS loudspeakers. Buyers would come in predisposed to listen to a JBL speaker vs maybe an AR speaker, or a Bose speaker vs an Infinity speaker. If the salepeople suggested listening to a JBL speaker vs an ESS speaker, the potential customer would often change the second contender to one of the other brands. ESS was confident they could sell their product against any other brand but they needed to give the market a reason to even listen to them, and their technology didn't seem to be motivating consumers enough. So at the big meetings with the ad agency about what sort of ad campaign would convince consumers to at least listen to ESS loudspeakers, it was determined that just using repetative ads wouldn't do it. ESS had already had a high frequency of ads. Since consumers said they gave JBL a listen due to their belief that pros used JBL so perhaps JBL made the consumer product they wanted. Consumers also said that as the inventor of the acoustic suspension or direct reflecting or "whatever" loudspeaker, other companies had some credibility with consumers. So the ad agency proposed that ESS go out and do a sextuple blind listening test nationwide with thousands of consumers under tightly controlled conditions, to get documented evidence that ESS loudspeakers sounded better in various price ranges compared to 9 other brands of loudspeakers. This was to be the "credibility hook" ESS supposedly needed to have consumers give them a listen. The Physics, Psychology/psychoacoustics, Audiology and Music departments at 4 major universities in California, Washington state, Wisconsin and Georgia would check the test controls and an outside accounting firm was brought in to tabulate and document the results. Of course ESS wanted to keep as much of the information gained as possible to themselves for future product development. But the gamble was that if their products were sonically superior, they could create an ad campaign that would convince more consumers to give their speakers a listen in retail stores. They gave me the job of actually running the test and managing all the different groups involved. I was chosen because the test was supposed to use the highest quality source material of classical, jazz, rock and pop music and some natural sounds. This meant getting access to original master tapes used to cut master pressing disks. I had the expertise needed to work with the studio people to make the part of the test program material using these tapes, I could manage a touring "road show" and I knew how to use quality test equipment to replicate the test conditions in each location. The musical passages had to be long enough to have some kind of repetative or sustained characeristics that would allow listeners to hear differences in loudspeakers. The program was 40 minutes long and covered every kind of musical material (although there was quite a bit of well recorded vocals in nearly half of the recordings). The speakers were all set up behind acoustically transparent screens that were visually opaque. About half of the listeners were in the near field. Almost every seat in the listening area was in a "sweet" spot. The colleges did allot to promote the participation of their students, staff and anyone who was interested, in newspapers and local radio. There were four different programs and there were four different price levels of speakers, although the programs were rotated between the different price levels. Consumers could come back to take the "test" up to three times if they so desired. Prizes were given out via drawings and the various departments at the universities were allowed to share some of the data collected at their facility. The playback system was never driven to clipping, the level of the balancing pre-amps was set for each speaker based on their average output using pink noise and flat, A, B and C weighted measurements both in the near and far fields. The locations of the speakers on stands was varied and with 9 speakers of which only 6 could be tested at a time even the comparison match-ups were constantly varied. There were tens of thousands of comparisons made by thousands of listeners. Quite a few audio business theories and guesses were tested in these circumstances. The program material did make a difference in the test results. Two different loudspeakers that were compared with different program material might show different audience preferences depending on whether the program material was pop, jazz or classical, so it seemed that even the best loudspeakers were not always "best" for all kinds of music. Of course there were some good loudspeakers that always seemed to do well in head to head comparisons using any program material and others that were just bad and were disliked by most listeners no matter what the program material. Sometimes the room's acoustics influenced the test results so that if the heat in the room built up and the window's opening positions were changed, the test results could change. ESS was able to show in random blind testing that their products were sonically superior "enough" (statistically valid results 5-8x greater than the margin of error) so that the test results were about the equivalent of the in-store research done by the ad agency. An interesting device was the "Comparison Identifier", a large box in the front of the room which displayed a number for "one" speaker in comparison vs the number for the "other" speaker being tested. The box would produce a number between 1 and 9 but the speakers could be labeled with any number at any time even during one listening test session. The test was actually quite well designed so that no speaker had an advantage. A few of the really poor sounding speakers were weeded out after the first hundred or so testing/listening sessions and some cross price-level comparisons could then be made. The marketing campaign (ESS Wins on Campus) was introduced before the tour of test locations was completed and so in Wisconsin during Homecoming week and at Georgia Tech the crowds of listeners wanting to "take the test" were enormous. The information not used for marketing purposes involved how 2 non-ESS branded loudspeakers did compared to each other. But one very startling fact came to light and it forced the testing to stop and the ad campaign to change. The most prefered loudspeaker for all kinds of music was the same at every college. It was the same for any test and any type of program material. Unfortunately this obviously "most preferred" loudspeaker was not the most expensive ESS loudspeaker. It was one of the least expensive ESS loudspeakers. The design criteria for this model of loudspeaker was actually quite elegant and as a model it shows how a device could be purposely made so it might "beat a test". By the time I had watched listeners in thousands of tests I could have designed a loudspeaker that would have always won the test by a statistically significant margin. ESS wasn't happy that their most expensive loudspeaker was only the most preferred by listeners when compared to other company's expensive loudspeakers. It seemed that when people made choices without visual cues as to which speaker "should" be best, their ears guided them in the direction of the most accurate sound, and that wasn't always the most expensive or the biggest loudspeaker. In addition this "up-the-line superiority" wasn't the same for all companies. Whether by design or accident, some companies produced speakers that sounded better as the price increased, but other companies had results like ESS where the tests determined that some of their less expensive loudspeakers were actually their most accurate. It turned out that the corporate philosophy of ESS that was dictated in part by the requirement for all mainline models to use the Heil tweeter, worked against them in some ways. There was also the fact that if the test could be duplicated because it was controlled, then any company could eventually figure out the 4 or 5 keys factors in the listener's sonic decision and design a loudspeaker that would "beat the test". This was an obvious outcome because there really could only be one set of major sonic priorities that people use to decide which loudspeaker sounds more accurate than another. Perhaps the priorities would be slightly different so my 1, 2, 3, 4,and 5 sonic priorities are 2, 5, 1, 6 and 3 for you, but generally there are only so many criteria that determine "realistic sound". Our brains and ears have evolved over a few millions of years making life and death decisions based on what our ancestors heard. So our hearing developed in a certain way prioritizing certain acoustic criteria. What was also obvious was that no one speaker company in 1978 made speaker systems that incorporated the most possible of these criteria in their designs. There were also limitations imposed by the fact that all the tested loudspeakers were bookshelf models because that is what dominated the market at the time. Strangely though, larger floor standing cabinets have many many sonic factors going against them and so it is less expensive to make a more realistic sounding speaker if it is a small or bookshelf model. There is also the "monkey-wrench" factor that dictates that people will be more disposed to spend more money if the loudpseaker they are auditioning looks bigger and more imposing because that loudspeaker seem visually to be worth more money and our visuals are telling our brain that the larger loudspeaker should sound better than smaller loudspeakers. In point of fact, the larger loudspeaker will likely sound less accurate than if the same money was put into a smaller loudspeaker. But the fact that this listening/test research could have been used against its sponsor and the loudspeaker industry as a whole points up how a simple sextuple blind test could become so dangerous for the status quo in loudspeakers in general. This should all give people pause when discussing comparison testing. Is it possible to actually design a test so that the second item being listened to, is not immediately disadvantaged (or advantaged) by it's position in the test? Is there any reason for the manufacturers to support such a test or even to acknowledge the results as being valid for reasons of their own? Is it possible in certain circumstances to even hear sonic differences without resorting to a basic change in venue (like the need to use headphones)? In fact, the loudspeaker and the source material are acknowledged to be the links of the chain that have the most problems, so if there is no effort to actually define how to reduce the magnitude of distortons in these areas, how can any other portion of the system even be tested? As I said in "the emperor's clothes" thread, there have been less than perhaps 10 loudspeaker systems ever made that can realistically reproduce the voice, the piano and natural sounds "accurately". If there has been so little effort to make realistic sounding loudspeakers and program material, what does it matter if the other parts of systems are .1% improved or not? Finally I have another anecdote that actually applies to me and what I want out of a listening test for my audiophile system. In the 80s when the compact disc was getting a firm foothold in audio, AES was concerned enough about what the 18kHz brick wall filter would sound like to try to develop an international listening test to see what kind of sonic impact these filters could have. They developed a test using playback systems that had lots of response output up to 50KHz, the way the Panasonic leaf tweeter could provide and then using playback material recorded with a sampling rate way past 80KHz they tested AES members to see how easily they could detect the insertion of filters of various types at various frequencies while music or real life sounds were being played. The idea was to push a button when you heard a filter inserted and release it when the filter was removed. At 3KHz everything sounded like it was playing over a telephone when the filter was inserted so that was easy. At 10KHz almost everybody noticed filter insertion. At 15 and 18KHz there were still many many engineers who noticed the filters. But at 23KHz I for one have only been able detect a filter about 20% of the time and at 25KHz I didn't hear any differences. But there was one young European recording engineer who kept getting it right all the time well past 20KHz. He was able to focus in on the hiss from the condenser microphone and pre-amp, plus the noise from the mixing console and the dither in the recording while excluding the other sounds in other spectra and he really could hear when these very high frequency filters were inserted and removed. That makes a total of one guy. I am absolutely certain that I would not want that one guy to ever make a decision (for me) in a test about any audio component item because I cannot hear what he hears so I don't want the decision about what is "best" to be made on the basis of any crieria except the ones I can hear. I want a test to determine anything in audio to be so repeatable and available that I can take the test myself because I don't want to base what will be best for me on what someone else can hear or someone else's taste in music. This is one of the things I learned by running a test taken by thousands of people. We all don't hear the same. We all don't make judgements on what sounds best using the same program material. Our sonic priorities for phase/square wave response, frequency response, dynamic reproduction capability, the intrusion of spurious cabinet noises and constant directivity might be similar but what is vastly different is what we DON'T hear. If I can't hear something then it isn't worth me paying for it in a product. I am a fussy listener compared to most and I am a trained audio engineer who can focus on individual specta and instruments while excluding others. But even so I don't want to take someone else's word for what's best even if they are similarly trained , because someone else will either have better or worse hearing than me, and I'll be the one spending the money on the equipment. Also whatever test will be developed to make the decisions about what is best, I want to be able to take it. But in case I didn't mention it, I'd rather not pay for the test itself because I know from experience, just how incredibly expensive this will all be. Paying for credible comparative listening tests is the part I haven't quite worked out yet. By the way, I've spoken to Ed Meitner about sonics many times in the past and I'm sure that he could easily produce the source player, turn-down/turn-up switches (we wouldn't want to make full power switches between products would we?) and level balancing pre-amp circuits needed for the kinds of test people here discuss, but, of course there would be a price to pay and I wouldn't want to be the one paying that price. Watchking Listening isn't a competative sport, but buying equipment is. We don't get enough sand in our glass |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Forum | |||
audio coax cable | High End Audio | |||
Equation for blind testing? | Audio Opinions | |||
DBT and science | High End Audio | |||
Acoustically transparent but opaque material for blind speaker testing? | General |