Home |
Search |
Today's Posts |
|
#1
|
|||
|
|||
Carey Carlan wrote:
"William Sommerwerck" wrote in : Last week at the Home Entertainment Show in New York Arny Krueger participated in a panel discussion with John Atkinson, editor of Stereophile magazine. Arny is well known for his support for the scientific method to test what is audible and what is not. The scientific method is not foolproof. Of course not, science is still based on human reason. Human reason is fallible. If human reason were perfect, the need to do experiments would drop precipitously. Simply removing certain obvious forms of bias does not mean the test results are accurate or are correlated to what we "actually" hear when we sit down to listen. Yes, it's incumbent on experimenters to do reasonable experiments. Absolutely correct. The scientific method systematically proves or disproves a single fact at a time within the limits of the control factors. More correctly, the analysis of scientific experiments confirm or deny hypothesis. However, it is possible to confirm or deny more than one hypothesis at a time with just one experiment. It cannot be extended to say that no remaining untested factors exist. Yes, it's incumbent on experimenters to do reasonable experiments. I will put my faith in those facts proven by this method, and leave the rest to wallow in the "snake oil" category. We should not forget that Sterephile has affirmed postmodernist anti-science. Given that John Atkinson has a degree in Physics, it's safe to say that he has a fair idea of what science is, and has chosen to go some other way. |
#2
|
|||
|
|||
Given that John Atkinson has a degree in Physics, it's safe
to say that he has a fair idea of what science is, and has chosen to go some other way. Not necessarily. Just because you've studied physics doesn't mean you've learned anything about the philosophy of science or how to conduct a proper experiment. Remember, The Stereophile began as a magazine that stood on the side of accurate (rather than euphonic) sound reproduction, and would help readers make better buying decisions by telling them what the major magazines didn't -- that there were often large and significant differences in the sound of otherwise similar-seeming components. In my opinion, * amps and preamps of 40+ years ago, both tube and transistor, were of lower absolute quality than those of today, and varied much more in subjective character. These differences were great enough to swamp the "perceptual noise" inherent in subjective testing. As a result, JGH's reviews were probably close to the truth (if such can be determined). I would like to see anyone with faith in ABX testing to round up some classic amplifiers sold before, oh, 1975, make sure they're in proper working order, and run ABX tests against either modern amps or a bypass. I'm reasonably confident that there _will_ be audible differences, and these will be similar to those reported by reviewers of that era. If so, it would greatly increase my trust in the validity of ABX testing in which no differences are heard. * Please note the IMO before you start jumping down my throat. |
#3
|
|||
|
|||
William Sommerwerck wrote:
Given that John Atkinson has a degree in Physics, it's safe to say that he has a fair idea of what science is, and has chosen to go some other way. Not necessarily. Just because you've studied physics doesn't mean you've learned anything about the philosophy of science or how to conduct a proper experiment. Given John's dabbling with postmodernist philosophy, point well taken. Remember, The Stereophile began as a magazine that stood on the side of accurate (rather than euphonic) sound reproduction, and would help readers make better buying decisions by telling them what the major magazines didn't -- that there were often large and significant differences in the sound of otherwise similar-seeming components. Agreed. JGH left Stereophile, didn't he? In my opinion, * amps and preamps of 40+ years ago, both tube and transistor, were of lower absolute quality than those of today, and varied much more in subjective character. I think that is a fact. Measure 'em, listen to 'em, they were as a rule, not all that good. There were some good ones but you got to pay for them. These differences were great enough to swamp the "perceptual noise" inherent in subjective testing. As a result, JGH's reviews were probably close to the truth (if such can be determined). I agree. IMO, the current naive audiophile belief that everything sounds different is a hangover from the days when everything did sound different. I would like to see anyone with faith in ABX testing to round up some classic amplifiers sold before, oh, 1975, make sure they're in proper working order, and run ABX tests against either modern amps or a bypass. We did that, but with an amp that was somewhat older than that. http://www.pcavtech.com/abx/abx_pwr.htm This page shows 4 amps that sounded the same, and 3 that didn't. I'm reasonably confident that there _will_ be audible differences, and these will be similar to those reported by reviewers of that era. If so, it would greatly increase my trust in the validity of ABX testing in which no differences are heard. Enjoy! |
#4
|
|||
|
|||
Ethan Winer wrote: Folks, Last week at the Home Entertainment Show in New York Arny Krueger participated in a panel discussion with John Atkinson, editor of Stereophile magazine. Arny is well known for his support for the scientific method to test what is audible and what is not. John is known for, um, - well, let's just call it an anti-science bias. You can read about the discussion and also download an MP3 file (30 MB, 1 hour long) he www.stereophile.com/news/050905debate/ Way to go, Arny! --Ethan I listened to most of this (as a part-time engineer, a subscriber to Skeptical Inquirer, and a long-time user of some of your software products). Clearly, if one can't tell the difference between two pieces of equipment, then, for the purpose of that listener, the two pieces of equipment are identical under those circumstances. Those who criticise DBT testing on general principles are on such non-scientific ground that they might as well join a church. What I don't get, and what I thought that Atkinson was getting at until he veered off into mysticism, is why the tests have to be conducted with short pieces of sound. If Atkinson's claim is that he can differentiate between different power amps when listening to them for an extended period, then let's design an experiment that tests this hypothesis, but remains double blind. How long does he need? A half hour on each? Ten minutes? An hour? Shouldn't be difficult - certainly, far more time has been spent arguing over this than would be necessary to conduct a *scientific* experiment as to whether two pieces of equipment can be differentiated under these circumstances. Kudos to Arny, indeed, for perservering when most others would have given up. I have long taken the view that the more idiots there are in the world, the better it is for me, so I don't try to educate them. I might even sell them some $2,500 power cables for them to plug into the Romex cable feeding their power outlets. |
#6
|
|||
|
|||
On Thu, 12 May 2005 05:46:03 +0000 (UTC), Steven Sullivan
wrote: wrote: ... If someone *passes* a DBT using longer samples, though, there's no basis for challenging the result due to the sample length. I think most objectivists would be 'OK' with such a report. There's no 'rule' that says the samples have to be short. If Atkinson's claim is that he can differentiate between different power amps when listening to them for an extended period, then let's design an experiment that tests this hypothesis, but remains double blind. How long does he need? A half hour on each? Ten minutes? An hour? Shouldn't be difficult - certainly, IIRC he lived with that transistor amp he hated for six months. far more time has been spent arguing over this than would be necessary to conduct a *scientific* experiment as to whether two pieces of equipment can be differentiated under these circumstances. I think the idea is that 'living with' the amps revealed a difference, that doing an DBT while unfamiliar with the amps didn't show. Fine! By all means, let's see if it made a difference. Now having formed quite definite 'feelings' about the difference in sound..to the point of being *sure* that one sounds better than another -- do another DBT. Should be easy to pass if you're right! And if so, you have valid grounds to start agitating for long 'acclimation' periods before doing DBTs of audio stuff. (Actually, researchers already routinely recommend and employ pre-DBT training sessions to sensitize the testees to differences... Arny K also recommends this on his pcabx site) If it takes long-term listening to show the difference, then do a test using long-term listening. How about having a large locked box (large enough for heat buildup not to be a problem) in the listening room that contains the amplifier. Once a day a 'maintenance person' comes in, and without the testee seeing (send him to the shower or something), opens up the box, spends five minutes doing something, tests the system so see that it works, locks the box, notifies the listener that he is through, and leaves until the next day. The listener then has 23 hours and 55 minutes of listening time until the maintenance person comes in again. During each visit, the maintenance man might or might not have changed out the amplifier (all amps used are precisely gain-matched). He might swap it at every visit for a week, then go two weeks just checking at each visit to see if the unit functions ok (off-site records are kept of what amp is in the box when). This should give adequate listening time at least for the listener to decided "like it" or "hate it" as Atkinson said of the transistor amp he had for several months that had passed a DBT (been indistinguishable from another good amp). With that sort of time frame (especially several weeks at a time) he should be able to say when the amp has been changed out. Sorry, with the above, the maintenance person knows what's what. Leave both amps in the locked box, make a switchbox driven by a microcontroller, the maintainer turns a keyswitch to activate it, it switches (inaudibly of course) or not, and displays a five-digit number that encodes the switch setting, the maintainer writes it down but doesn't know how to decode it into a switch setting. For longterm listening (several days of one amp at a time), make the chances of switching much less than 1/2. So it's a good DBT. But then, what was that Presidential quote, "We have nothing to fear but truth itself..." And now I wonder why I just spent my time typing all that... ----- http://mindspring.com/~benbradley |
#7
|
|||
|
|||
Or for that matter an automated system that switches (or not) between
units. It could be run from a PC with a form to fill out with your opinions and ratings of the days listening. Pretty simple to implement. Serial control of a couple good relays. You could test any reasonable number of amps in a month. You could even make it interactive. As long as the tester has no clue as to which amp it being listened to at any time. Just ABC&D in a sealed black box and some sort of switching. |
#8
|
|||
|
|||
Clearly, if one can't tell the difference between two pieces of
equipment, then, for the purpose of that listener, the two pieces of equipment are identical under those circumstances. Exactly... "Under those circumstances." Double-blind testing, as it is currently implemented, is not equivalent to simply sitting down and listening to music. Nor is "subjective" testing, for that matter. Those who criticise DBT testing on general principles are on such non-scientific ground that they might as well join a church. Not at all. Calling something "scientific" does not make it so. (The word itself implies a degree of "truthfulness" that is not fully justified.) Simply because double-blind testing is useful in other areas does not mean it provides useful or valid results when judging hi-fi equipment. What most people conveniently ignore when criticizing my views is that I don't agree with either side in this issue. Both sides are "wrong," because their testing procedures have not been proven to be correct. Simply removing bias does not guarantee accurate, valid, or useful results. What I don't get, and what I thought that Atkinson was getting at until he veered off into mysticism, is why the tests have to be conducted with short pieces of sound. If Atkinson's claim is that he can differentiate between different power amps when listening to them for an extended period, then let's design an experiment that tests this hypothesis, but remains double blind. How long does he need? A half hour on each? Ten minutes? An hour? Shouldn't be difficult - certainly, far more time has been spent arguing over this than would be necessary to conduct a *scientific* experiment as to whether two pieces of equipment can be differentiated under these circumstances. What is needed -- and I could name several well-known people who agree with me -- is long-term blind listening tests in which people simply sit down and listen for pleasure. Properly conducted, such testing would would provide useful information about "how" people listen, what they think they hear, and establish a baseline for judging "subjective" and "objective" testing. But such testing would require many listeners, take a lot of time, and be difficult to implement and run correctly. Not to mention the fact that both subjectivists and objectivists have a vested interest in believing what they want to believe. People are uncomfortable changing their world views. Kudos to Arny, indeed, for perservering when most others would have given up. I have long taken the view that the more idiots there are in the world, the better it is for me, so I don't try to educate them. I might even sell them some $2,500 power cables for them to plug into the Romex cable feeding their power outlets. |
#9
|
|||
|
|||
William Sommerwerck wrote:
What is needed -- and I could name several well-known people who agree with me -- is long-term blind listening tests in which people simply sit down and listen for pleasure. Properly conducted, such testing would would provide useful information about "how" people listen, what they think they hear, and establish a baseline for judging "subjective" and "objective" testing. But such testing would require many listeners, take a lot of time, and be difficult to implement and run correctly. Not to mention the fact that both subjectivists and objectivists have a vested interest in believing what they want to believe. People are uncomfortable changing their world views. FWIW most if not all the original ABX partners did exactly what is described here. They picked out two components to compare, did long-term ABX testing, and compared their results to shorter term tests. There have also been some more-formal tests that David Clark did with I think it was Larry Greehill. Bottom line - no joy from the long term tests. If you can't hear a difference in a well-done short term test, listening for hours, days or weeks per trial hasn't been found to help. In fact, long trials can be shown to hurt listener sensitivity, because they temporally displace the listening experiences being compared even more, and that is known to be a bad thing. |
#10
|
|||
|
|||
FWIW most if not all the original ABX partners did exactly
what is described here. They picked out two components to compare, did long-term ABX testing, and compared their results to shorter term tests. There have also been some more-formal tests that David Clark did with I think it was Larry Greehill. That's not at all what I'm suggesting. The listeners would simply be relaxing, playing their favorite music, without any knowledge of the electronics in use, and without any attempt to make distinctions. In other words, we simply want to know what they think they hear. After a few months (!!!), components might be substituted -- without the listeners' knowledge -- to see how they react. In fact, long trials can be shown to hurt listener sensitivity, because they temporally displace the listening experiences being compared even more, and that is known to be a bad thing. Agreed (more or less). But that's one of the reasons for running such a test -- to see how such things change. |
#11
|
|||
|
|||
On Thu, 12 May 2005 15:36:50 -0700, "William Sommerwerck"
wrote: and without any attempt to make distinctions. Another biscuit crux. The *attempt* itself is contaminating. We might not (I would say experientially *do not*) listen/ hear the same for enjoyment as for "testing". An oft-observed fact is that eye witnesses to catastrophic events are amazingly unreliable. We're bred to interpret the world through a maze of models, assumptions and imagination. This discussion is about those things; let's just not forget the "bred" part's true relevance. Good fortune, Chris Hornbeck |
#12
|
|||
|
|||
William Sommerwerck wrote:
FWIW most if not all the original ABX partners did exactly what is described here. They picked out two components to compare, did long-term ABX testing, and compared their results to shorter term tests. There have also been some more-formal tests that David Clark did with I think it was Larry Greehill. That's not at all what I'm suggesting. The listeners would simply be relaxing, playing their favorite music, without any knowledge of the electronics in use, and without any attempt to make distinctions. Huh? That is what we did! In other words, we simply want to know what they think they hear. Been there done that. After a few months (!!!), components might be substituted -- without the listeners' knowledge -- to see how they react. Tell you what William, if you can get anybody with a life to play by these rules, give me a call. In fact, long trials can be shown to hurt listener sensitivity, because they temporally displace the listening experiences being compared even more, and that is known to be a bad thing. Agreed (more or less). But that's one of the reasons for running such a test -- to see how such things change. We did it and it kinda left this bad taste in our mouths. Null results from lont-term listening when quick switching gives positive results can do that to a person. |
#13
|
|||
|
|||
Arny Krueger wrote:
William Sommerwerck wrote: What is needed -- and I could name several well-known people who agree with me -- is long-term blind listening tests in which people simply sit down and listen for pleasure. Properly conducted, such testing would would provide useful information about "how" people listen, what they think they hear, and establish a baseline for judging "subjective" and "objective" testing. But such testing would require many listeners, take a lot of time, and be difficult to implement and run correctly. Not to mention the fact that both subjectivists and objectivists have a vested interest in believing what they want to believe. People are uncomfortable changing their world views. FWIW most if not all the original ABX partners did exactly what is described here. They picked out two components to compare, did long-term ABX testing, and compared their results to shorter term tests. There have also been some more-formal tests that David Clark did with I think it was Larry Greehill. Nousaine has also conducted 'long term' tests, where subjects were allowed to acclimate themselves for *weeks* before actually taking the test. -- -S It's not my business to do intelligent work. -- D. Rumsfeld, testifying before the House Armed Services Committee |
#14
|
|||
|
|||
"Steven Sullivan" wrote in message ... Arny Krueger wrote: William Sommerwerck wrote: What is needed -- and I could name several well-known people who agree with me -- is long-term blind listening tests in which people simply sit down and listen for pleasure. Properly conducted, such testing would would provide useful information about "how" people listen, what they think they hear, and establish a baseline for judging "subjective" and "objective" testing. But such testing would require many listeners, take a lot of time, and be difficult to implement and run correctly. Not to mention the fact that both subjectivists and objectivists have a vested interest in believing what they want to believe. People are uncomfortable changing their world views. FWIW most if not all the original ABX partners did exactly what is described here. They picked out two components to compare, did long-term ABX testing, and compared their results to shorter term tests. There have also been some more-formal tests that David Clark did with I think it was Larry Greehill. Nousaine has also conducted 'long term' tests, where subjects were allowed to acclimate themselves for *weeks* before actually taking the test. That has nothing at all to do with taking the test, per se. |
#15
|
|||
|
|||
William,
What is needed ... is long-term blind listening tests I have to agree with Arny. I can't see why listening long term increases someone's ability to discern small differences. If anything I'd say it's the other way around. But even if that were so, if you have to listen for a month to detect some tiny improvement, how important really is that improvement? I know that when I A/B stuff where the differences are very small (not blind, just fooling around) I need to hear the exact same short passage over and over. A friend once asked me to listen for a change in a song his client sent out for mastering. The ME claimed he made it "better" but my friend couldn't hear any difference. I couldn't either, but I also "couldn't tell if I could tell" until we took both versions of the tune and lined them up in his DAW. Before we did that, one version might be playing a verse while the next was at the chorus. Just having a different chord was enough to throw off any perception of low end clarity and fullness from one version to the next. But once I set up each tune to play the exact same 5 second passage - over and over while switching back and forth - I was then able to conclude with certainty that there was no meaningful difference. Also, it is well known that the ear adjusts to changes in sound pretty easily. So if anything, long term listening (live with a new power cable for a month) will tend to *mask* real differences rather than reveal them better. --Ethan |
#16
|
|||
|
|||
Ethan Winer wrote:
William, What is needed ... is long-term blind listening tests I have to agree with Arny. I can't see why listening long term increases someone's ability to discern small differences. In a way it does. Sometimes you have to listen a long time before you set the stage for the audible difference to be maximually audible. Stuff like a certain rim shot, etc. |
#17
|
|||
|
|||
On Thu, 12 May 2005 12:02:10 -0400, "Ethan Winer" ethanw at
ethanwiner dot com wrote: I have to agree with Arny. I can't see why listening long term increases someone's ability to discern small differences. If anything I'd say it's the other way around. ... I know that when I A/B stuff where the differences are very small (not blind, just fooling around) I need to hear the exact same short passage over and over. ... Also, it is well known that the ear adjusts to changes in sound pretty easily. So if anything, long term listening (live with a new power cable for a month) will tend to *mask* real differences rather than reveal them better. When I was designing equipment (for myself), and struggling daily with the question of whether a tiny improvement was real, I came to the following conclusion: My brain is willing to suspend disbelief in any halfway decent electronically created illusion of sound for about thirty seconds. After that, it quickly begins removing trust in those aspects of the illusion that are not sufficiently well reproduced. If I can A/B for thirty seconds back and forth, I can (if there is a difference) hear the increase and decrease in realism. (For awhile, then fatigue sets in.) If I listen to the less realistic sample for a minute or more, my brain disables my ability to trust in those aspects of the illusion that were changing. Both samples now sound the same, because I'm no longer listening for those differences. It may take a half-hour or more of listening to only the more realistic configuration (or only real, not reproduced sound) before I can trust whatever aspect of illusion was being varied. So the procedure was listen for half an hour, make a change, and decide within thirty seconds whether there was a decrease in realism. If there was a decrease, go ahead and repeat the test in the other direction, but don't be surprised when there is no audible increase in realism. Am I the only one whose brain works this way? Loren |
#18
|
|||
|
|||
When I was designing equipment (for myself), and struggling daily with
the question of whether a tiny improvement was real, I came to the following conclusion: My brain is willing to suspend disbelief in any halfway decent electronically created illusion of sound for about thirty seconds. After that, it quickly begins removing trust in those aspects of the illusion that are not sufficiently well reproduced. If I can A/B for thirty seconds back and forth, I can (if there is a difference) hear the increase and decrease in realism. (For awhile, then fatigue sets in.) If I listen to the less realistic sample for a minute or more, my brain disables my ability to trust in those aspects of the illusion that were changing. Both samples now sound the same, because I'm no longer listening for those differences. It may take a half-hour or more of listening to only the more realistic configuration (or only real, not reproduced sound) before I can trust whatever aspect of illusion was being varied. So the procedure was listen for half an hour, make a change, and decide within thirty seconds whether there was a decrease in realism. If there was a decrease, go ahead and repeat the test in the other direction, but don't be surprised when there is no audible increase in realism. Am I the only one whose brain works this way? I doubt that you're unique, but the real issue is whether the differences you think you hear really do exist. You are assuming that because you think you hear a difference, you really do. You can't assume that, any more than those supporting double-blind testing can assume it gives correct and complete results. |
#19
|
|||
|
|||
What is needed ... is long-term blind listening tests
I have to agree with Arny. I can't see why listening long term increases someone's ability to discern small differences. I guess I'm not explaining things clearly enough to overcome your preconceptions. The purpose of long-term blind listening is not (initially) to make distinctions, but to simply see how we listen, and how we react to a particular system. For example, if the system remains unchanged, but people report differences in its sound (especially if different people report different differences), then we start to have an idea, of the character and magnitude of what I call "perceptual noise". This would be useful to know, as it has a significant effect on subjective testing, and (I think) at least a little on ABX and similar methodologies. |
#20
|
|||
|
|||
William Sommerwerck wrote:
What is needed ... is long-term blind listening tests I have to agree with Arny. I can't see why listening long term increases someone's ability to discern small differences. I guess I'm not explaining things clearly enough to overcome your preconceptions. The problem is Arny thinks people's auditory memory is about 1/10 of a second. I dunno, maybe his is. I can remember things I heard 40 years ago. |
#21
|
|||
|
|||
On Thu, 12 May 2005 15:43:34 -0700, "William Sommerwerck"
wrote: For example, if the system remains unchanged, but people report differences in its sound (especially if different people report different differences), then we start to have an idea, of the character and magnitude of what I call "perceptual noise". This would be useful to know, as it has a significant effect on subjective testing, and (I think) at least a little on ABX and similar methodologies. It does make a lot of sense to work at establishing a noise floor first. Glad to hear you're still working on the project; your new posts sound very positive. Chris Hornbeck |
#22
|
|||
|
|||
William,
The purpose of long-term blind listening is ... to simply see how we listen, and how we react to a particular system. Okay, fine. But long term, like a month, includes effects such as: * Your spouse is in a bad mood lately and bugs you all the time with petty complaints. * The restaurant where you eat lunch every day hired a new chef who uses more garlic than the last chef. * One of your kids just got admitted to the college she wanted, but it's going to cost you twice as much as the college *you* wanted her to go to. And so forth. And Yes, I am dead serious with these examples, and I'm sure there are many more in that vein. For balance you can also add a few "positive" life changes to the negative ones I listed. Like you finally got the big promotion you've worked so hard for, and it comes with a $10k salary increase. Yep, those new speaker cables are starting to sound mighty good now that they've finally "broken in." if the system remains unchanged, but people report differences Yes, I'm sure this is the biggest factor. It's why so many otherwise intelligent people think a new power cord made a difference. It's why even a pro mix engineer can sometimes tweak a kick drum EQ to perfection, only to discover later he was adjusting the rhythm guitar track. I've made this point before, and it needs to be made repeatedly: One of the things that astounds me is how audiophiles - and especially magazine reviewers - claim to be able to discern tiny changes while listening in a room where fully half of the SPL is dominated by ambience and early reflections. When I read a reviewer comment on a particular loudspeaker's imaging, and I *know for a fact* that the reviewer has no acoustic treatment at all, I have to dismiss everything else from that reviewer. And a lack of even minimal acoustic treatment probably dismisses 95 percent of all audio reviewers, no? --Ethan |
#23
|
|||
|
|||
"Ethan Winer" ethanw at ethanwiner dot com wrote:
[...] I can't see why listening long term increases someone's ability to discern small differences. If anything I'd say it's the other way around. I dunno... When I first heard the Neumann KM184 I thought it might be the best mic ever. The more I used it, the more I liked it. Then, over time, I discovered some of its weaknesses. I also got past the Honeymoon and realized that the top-end bloom was "exciting" at first, but might be kind of annoying after a while. I also thought the RNC was pretty lame the first time I tried it. The more I used it though, the more I realized that what makes it a great tool is the absence of characteristics that stand out in a short-term test. I can think of two or three more examples, but you get the point. Getting familiar with a device can be really helpful in making valid decisions about how it sounds. -- "It CAN'T be too loud... some of the red lights aren't even on yet!" - Lorin David Schultz in the control room making even bad news sound good (Remove spamblock to reply) |
#24
|
|||
|
|||
Lorin,
Then, over time, I discovered some of its weaknesses. I can't disagree with anything you said. But that is very different from the notion you can't distinguish one piece of gear from another *at all* unless you live with it for a while. Those on the "other side" of this argument are saying two power amps might sound the same in an A/B test, but after some extended time they can then hear a difference. Of course, they're not even saying they can hear a difference in an A/B test only after a while. They're saying, well, I guess I don't know what they're saying. But in your case I'm sure you could hear the difference immediately between one microphone and another, or one compressor and another. What you're talking about is learning to appreciate - or not - the *character* of some device over time. And to me this is very different from being able to detect any change at all. --Ethan |
#25
|
|||
|
|||
William Sommerwerck wrote: Clearly, if one can't tell the difference between two pieces of equipment, then, for the purpose of that listener, the two pieces of equipment are identical under those circumstances. Exactly... "Under those circumstances." Double-blind testing, as it is currently implemented, is not equivalent to simply sitting down and listening to music. Nor is "subjective" testing, for that matter. Those who criticise DBT testing on general principles are on such non-scientific ground that they might as well join a church. Not at all. Calling something "scientific" does not make it so. (The word itself implies a degree of "truthfulness" that is not fully justified.) Simply because double-blind testing is useful in other areas does not mean it provides useful or valid results when judging hi-fi equipment. What most people conveniently ignore when criticizing my views is that I don't agree with either side in this issue. Both sides are "wrong," because their testing procedures have not been proven to be correct. Simply removing bias does not guarantee accurate, valid, or useful results. What I don't get, and what I thought that Atkinson was getting at until he veered off into mysticism, is why the tests have to be conducted with short pieces of sound. If Atkinson's claim is that he can differentiate between different power amps when listening to them for an extended period, then let's design an experiment that tests this hypothesis, but remains double blind. How long does he need? A half hour on each? Ten minutes? An hour? Shouldn't be difficult - certainly, far more time has been spent arguing over this than would be necessary to conduct a *scientific* experiment as to whether two pieces of equipment can be differentiated under these circumstances. What is needed -- and I could name several well-known people who agree with me -- is long-term blind listening tests in which people simply sit down and listen for pleasure. Properly conducted, such testing would would provide useful information about "how" people listen, what they think they hear, and establish a baseline for judging "subjective" and "objective" testing. But such testing would require many listeners, take a lot of time, and be difficult to implement and run correctly. Not to mention the fact that both subjectivists and objectivists have a vested interest in believing what they want to believe. People are uncomfortable changing their world views. I don't know why you think that objectivists are against changing their views. I certainly consider myself one, but if someone demonstrated that they could consistently distinguish two power cords then I would believe that there were audible differences between them. It is easy enough to design an experiment to do this - it only takes 1% of the effort that has been spent arguing about it. "Objectivists" (or "rationalists", as I would call them) do not believe that there are no differences between components, so we don't have any "world views" to be uncomfortable changing. I believe very strongly that there are big differences between speakers and microphones, for instance. It is not part of my weltanschuung (sp?) that there are no differences between good power amps. I believe that there are audible differences between mic pre amps, so I don't see why there wouldn't be audible differences between power amps. But until people can distinguish good power amps there's no reason to suppose that they sound different. Kudos to Arny, indeed, for perservering when most others would have given up. I have long taken the view that the more idiots there are in the world, the better it is for me, so I don't try to educate them. I might even sell them some $2,500 power cables for them to plug into the Romex cable feeding their power outlets. |
#26
|
|||
|
|||
"William Sommerwerck" wrote in message ... Clearly, if one can't tell the difference between two pieces of equipment, then, for the purpose of that listener, the two pieces of equipment are identical under those circumstances. Exactly... "Under those circumstances." Double-blind testing, as it is currently implemented, is not equivalent to simply sitting down and listening to music. Nor is "subjective" testing, for that matter. Those who criticise DBT testing on general principles are on such non-scientific ground that they might as well join a church. Not at all. Calling something "scientific" does not make it so. (The word itself implies a degree of "truthfulness" that is not fully justified.) Simply because double-blind testing is useful in other areas does not mean it provides useful or valid results when judging hi-fi equipment. What most people conveniently ignore when criticizing my views is that I don't agree with either side in this issue. Both sides are "wrong," because their testing procedures have not been proven to be correct. Simply removing bias does not guarantee accurate, valid, or useful results. What I don't get, and what I thought that Atkinson was getting at until he veered off into mysticism, is why the tests have to be conducted with short pieces of sound. If Atkinson's claim is that he can differentiate between different power amps when listening to them for an extended period, then let's design an experiment that tests this hypothesis, but remains double blind. How long does he need? A half hour on each? Ten minutes? An hour? Shouldn't be difficult - certainly, far more time has been spent arguing over this than would be necessary to conduct a *scientific* experiment as to whether two pieces of equipment can be differentiated under these circumstances. What is needed -- and I could name several well-known people who agree with me -- is long-term blind listening tests in which people simply sit down and listen for pleasure. Properly conducted, such testing would would provide useful information about "how" people listen, what they think they hear, and establish a baseline for judging "subjective" and "objective" testing. But such testing would require many listeners, take a lot of time, and be difficult to implement and run correctly. Not to mention the fact that both subjectivists and objectivists have a vested interest in believing what they want to believe. People are uncomfortable changing their world views. Your points are in total agreement with the argument I have been carrying on in RAHE for the last 1 1/2 years. I've sketched out a "control test" with one phase exactly as you mention...actually first a sighted listening long term evaluative stage...then a blind stage otherwise identical, then a blind short-term evaluative stage in a neutral environment, and finally a short-term blind comparative stage in a neutral environment. This was designed to provide all of the bridges between long term listening for enjoyment all the way to conventional a-b or a-b-x testing as it is promoted and practiced by Arny and others. If the correlation broke down, we would know where and accordingly most probably why. If it didn't, it would convert most subjectivists to objectivists. The drawback: expensive, difficult to stage, time-consuming, and requiring several hundred people. Only one of the objectivists there would even consider such a test...most denied the need for any test. They basically state, as Arny did at the Stereophile show, that he knows dbt works because it gives the same audiometric results as previous blind tests. Talk about being impervious to the underlying assumptions...... |
#27
|
|||
|
|||
Your points are in total agreement with the argument I have been carrying on
in RAHE for the last 1 1/2 years. I've sketched out a "control test" with one phase exactly as you mention...actually first a sighted listening long term evaluative stage...then a blind stage otherwise identical, then a blind short-term evaluative stage in a neutral environment, and finally a short-term blind comparative stage in a neutral environment. This was designed to provide all of the bridges between long term listening for enjoyment all the way to conventional a-b or a-b-x testing as it is promoted and practiced by Arny and others. If the correlation broke down, we would know where and accordingly most probably why. If it didn't, it would convert most subjectivists to objectivists. The drawback: expensive, difficult to stage, time-consuming, and requiring several hundred people. Only one of the objectivists there would even consider such a test...most denied the need for any test. They basically state, as Arny did at the Stereophile show, that he knows dbt works because it gives the same audiometric results as previous blind tests. Talk about being impervious to the underlying assumptions...... It's nice that people are finally starting to understand what I'm talking about, and contributing good ideas of their own. |
#28
|
|||
|
|||
William Sommerwerck wrote:
It's nice that people are finally starting to understand what I'm talking about, and contributing good ideas of their own. Lavo? No way! I don't think he hears what others say at all. |
#29
|
|||
|
|||
Harry Lavo wrote:
They basically state, as Arny did at the Stereophile show, that he knows dbt works because it gives the same audiometric results as previous blind tests. Talk about being impervious to the underlying assumptions... Talk about distorting what you heard until it was what you want to hear. I said other means - that they were also blind tests would be yet another fabrication of your mind, Lavo. The good news Harry is that I was able to reduce your seemingly-endless post to just two fairly-brief sentences. |
#30
|
|||
|
|||
What is needed -- and I could name several well-known people who
agree with me -- is long-term blind listening tests in which people simply sit down and listen for pleasure. Properly conducted, such testing would would provide useful information about "how" people listen, what they think they hear, and establish a baseline for judging "subjective" and "objective" testing. I think this is an important point. DBTs are great, and inarguably valid from a certain point of view. But testers in these settings tend to listen "hard", with the analytical part of their minds and their sensory aparatus, and much less so with their intuitive, subjective, emotional side. It's arguable that since music in particular is generally consumed by listeners in the latter state, a rigidly "objective" analysis may miss something. It reminds me of some of my bad gear decisions through my life as a music/audio enthusiast. Back in the 70s, I conducted a serious search for better speakers to replace my very enjoyable but somewhat limited KLH 17s (these were relatively affordable 60s era 2 way acoustic suspension bookshelf speakers, well regarded but nothing particularly special or expensive). After exhaustive research including many, many hours of critical listening tests, dozens of magazine reviews and so on, I chose the Advent "Large" speakers (their first product). They sounded really wonderful to me, better than any of my other candidates, and had received unanimously glowing reviews in the audio press. I brought them home, set them up in place of the KLHs and prepared to be very pleased. Initially, as I "evaluated" my choice, they proved to be every bit as good as I had hoped; very wide frequency response, low distortion, excellent dispersion and so on. I was rockin! Or was I... After passing my post-purchase evaluation process with flying colors, of course the next thing was to just relax and enjoy music on them, and that's where it all started to go wrong. No sooner had I switched off my "objective", analytical mind when I began to realize I was no longer enjoying my favorite music as much. Something was interfering with the connection between the emotional intent of the musical performance and my sense of it. Very disturbingly, the magic was somehow gone from my favorite albums. I immediately isolated the speakers as the problem of course, because they were the only element that had changed. The curious thing though was that every time I put my analytical "hat" back on. the Advents simply blew the KLHs away in every single way, and were clearly excellent performers, as everyone else seemed to agree. Forget the analysis though, put the "enjoy music" hat back on and...big problem. I never got past it, and ultimately sold the Advents and went back to the KLHs. The magic came right back, and once again my favorite music could take me to the joyous, transcendant places it had before. Similar experiences happened other times with speakers and other hi-fi gear, enough to make me painfully aware of the pitfalls of "critical" listening. There may also be an element in DBTs that, as a side effect of their "objectivity", doesn't incorporate a level of sensory experience that *transcends* the objective, and could thereby have significant consequences in terms of judgements thus made. Offerered respectfully for your consideration... Ted Spencer, NYC |
#31
|
|||
|
|||
Ted:
I never got past it, and ultimately sold the Advents and went back to the KLHs. The magic came right back, and once again my favorite music could take me to the joyous, transcendant places it had before. You found your happy place. You were tuned to those speakers over time and they became your reference. Nothing wrong with that. My Genesis 22s have become my reference. There are better speakers out there I assume but they're pretty darn good and make me happy. I'm just glad there's a former Genesis employee out there still making and reconing the drivers or I'd have to change them. |
#32
|
|||
|
|||
wrote:
What is needed -- and I could name several well-known people who agree with me -- is long-term blind listening tests in which people simply sit down and listen for pleasure. Properly conducted, such testing would would provide useful information about "how" people listen, what they think they hear, and establish a baseline for judging "subjective" and "objective" testing. I think this is an important point. DBTs are great, and inarguably valid from a certain point of view. But testers in these settings tend to listen "hard", with the analytical part of their minds and their sensory aparatus, and much less so with their intuitive, subjective, emotional side. It's arguable that since music in particular is generally consumed by listeners in the latter state, a rigidly "objective" analysis may miss something. It reminds me of some of my bad gear decisions through my life as a music/audio enthusiast. Back in the 70s, I conducted a serious search for better speakers to replace my very enjoyable but somewhat limited KLH 17s (these were relatively affordable 60s era 2 way acoustic suspension bookshelf speakers, well regarded but nothing particularly special or expensive). After exhaustive research including many, many hours of critical listening tests, dozens of magazine reviews and so on, I chose the Advent "Large" speakers (their first product). They sounded really wonderful to me, better than any of my other candidates, and had received unanimously glowing reviews in the audio press. I brought them home, set them up in place of the KLHs and prepared to be very pleased. Initially, as I "evaluated" my choice, they proved to be every bit as good as I had hoped; very wide frequency response, low distortion, excellent dispersion and so on. I was rockin! Or was I... After passing my post-purchase evaluation process with flying colors, of course the next thing was to just relax and enjoy music on them, and that's where it all started to go wrong. No sooner had I switched off my "objective", analytical mind when I began to realize I was no longer enjoying my favorite music as much. Something was interfering with the connection between the emotional intent of the musical performance and my sense of it. Very disturbingly, the magic was somehow gone from my favorite albums. I immediately isolated the speakers as the problem of course, because they were the only element that had changed. The curious thing though was that every time I put my analytical "hat" back on. the Advents simply blew the KLHs away in every single way, and were clearly excellent performers, as everyone else seemed to agree. Forget the analysis though, put the "enjoy music" hat back on and...big problem. I never got past it, and ultimately sold the Advents and went back to the KLHs. The magic came right back, and once again my favorite music could take me to the joyous, transcendant places it had before. Similar experiences happened other times with speakers and other hi-fi gear, enough to make me painfully aware of the pitfalls of "critical" listening. There may also be an element in DBTs that, as a side effect of their "objectivity", doesn't incorporate a level of sensory experience that *transcends* the objective, and could thereby have significant consequences in terms of judgements thus made. Offerered respectfully for your consideration... I'l repost here what is relevent to this point *********************** Taste. What tastes bad? Suppose, when one is young, one eats something, but very shortly afterwards, gets ill. What can happen is that that it can trigger a dislike for that food. The body processing may attempt to link that food taste with "bad", even if the correlation was incorrect, it may still do this. There reason for this is that it is very difficult to have hardware, i.e. genes, deal with all possible variations in the environment. How does one know in advance that a certain taste should be perceived as good or bad? evolution solves this problem by making all perceptions and emotions, software programmable by the environment, i.e. memes. ***************** You have already been programmed as to what sounds "good" i.e. the KLHs. If you had gotten your first shag while listening to the Advents rather than the KLHs, things would have been different.:-) Kevin Aylward http://www.anasoft.co.uk SuperSpice, a very affordable Mixed-Mode Windows Simulator with Schematic Capture, Waveform Display, FFT's and Filter Design. |
#33
|
|||
|
|||
You have already been programmed as to what sounds "good" i.e. the
KLHs. If you had gotten your first shag while listening to the Advents rather than the KLHs, things would have been different.:-) Kevin Aylward Nah. I've spent my entire adult life as an audio professional. I've encountered many "new and different" pieces of gear that *do* rock my world musically, immediately, and only a few (thankfully) that fit the description I made earlier. It's not a matter of what I'm "programmed" to like. I appreciate a very wide spectrum of audio gear, and of music for that matter, and I know how to qualify what I hear and feel. Another case: there was a certain brand of very high end mic pre/eq/compressor I was invited to evaluate a couple of years ago, and it measured and "objectively analyzed" sensationally. It also got rave reviews in the pro audio press and by many here. It just never passed audio in a musical way to my ears. I was offered a *really* great price on this $3000 piece, which had been sent to me brand new, and I turned it down. It always made me feel like I was listening to *equipment*, not music. That's the best way I know how to put it. Trust me on this...please... Ted Spencer, NYC |
#34
|
|||
|
|||
"Kevin Aylward" wrote:
You have already been programmed as to what sounds "good" i.e. the KLHs. If you had gotten your first shag while listening to the Advents rather than the KLHs, things would have been different.:-) Interesting tangent, but essentially irrelevant to the point, which was: Ted's "analytical ears" preferred one, while his "kickin' back" ears preferred the other. The point is NOT that he preferred one or the other, but that his preference *varied* according to his listening "state." That has significant implications for how we evaluate audio equipment. Ted, I've had exactly the same experience, with opposite results. I bought a pair of speakers for my girlfriend, which I set up against my trusty old Energys. I thought mine sounded better, based on three or four points. Now, as I sit here in Kathy's living room just *listening* to the ones I bought for her (as opposed to "analyzing" them), I think they may be more natural overall than mine. Different mind set, different conclusion. Of course, it could be the room... ....and so it goes. -- "It CAN'T be too loud... some of the red lights aren't even on yet!" - Lorin David Schultz in the control room making even bad news sound good (Remove spamblock to reply) |
#35
|
|||
|
|||
Ted, one thing you might want to consider is that in the 1970s, you
presumably bought a phono cartridge that you liked the sound of when it was used with your KLH speakers. Change the speakers, and that choice of cartridge is no longer the same choice you would most likely have made. The large Advents had a more forward sound than any KLHs that I can recall from that period. A more insidious hidden issue is your selection of recordings that served as the touchstone for your emotional responses. Your KLH speakers (with your existing phono cartridge) let you enjoy recordings that, through less rolled-off sounding speakers, would have sounded too bright or perhaps too closely miked. That's probably good--most record producers like "hotter" sound than most record listeners, especially listeners who attend classical concerts and have any idea what real music sounds like in a hall. But by the time you've spent six months or a year with the KLH-based system, your roster of favorite recordings has been influenced by the combined sonic characteristics of your particular cartridge and loudspeakers. If you had started off with the Advents but used the same phono cartridge as with the KLHs, then assuming that you're open to a wide enough range of musical repertoire, you might have been more impressed with flashier recordings that the Advents were effective at showing off. For example, if you enjoyed listening to string quartets with the KLHs, the Advents might have had you humming along to Mahler symphonies or "The Planets." _Then_ if you switched to KLHs, you would miss the color and presence of the Advents, with very little to compensate for their loss, since the effectiveness of those recordings depends so much on that presence and color. A mellow presentation of dramatic large-scale pieces is a different taste that you might need to acquire. Do you see what I'm getting at here? --best regards |
#36
|
|||
|
|||
Thanks for your thoughtful reply. The problem is that, as I said, the
Advents passed all my "objective" listening criteria with flying colors. There was no overtly empirical issue at all such as things sounding "more forward" or "too bright or too closely miked". All those criteia were just splendid to my critical ear. And my cartridge by the way, if I remember correvtly, was a Shure V15-III, unless it was the Shure M92E that I had before it. In any case though, perhaps I can add a little more to the story about what bothered me. To begin with, my musical taste then, as now for the most part, revolved primarily around vocal pop, rock and folk-rock. I'm pretty sure I bought the Advents in 1973, so I would have been listening to James Taylor, The Eagles, Joni Mitchell, The Beatles, Neil Young, Hendrix, Blind Faith...you get the idea. Searching my sense memory for what the Advents lacked when I simply wanted to enjoy the music rather than "analyze the gear", I recall that it was a sense of transparency and realness in the midrange, especially on vocals - something I will say that the KLHs presented very well despite their limitations. While I'm working from long memory here, I'll say that the Advents (or at least my pair) had some kind of phase issue in the heart of the midrange, possibly to do with the crossover frequency or filter design (just a guess). The effect was what ultimately turned out to be a very distracting loss of clarity, focus and "integrity", if you will, of the center channel vocal image. Since my listening was highly focused on the vocal performance (it was after all the height of the "sensitive" singer/songwriter era), although not at all to the exclusion of other elements, this particular attribute was a deal-breaker for me. I just didn't feel the connection to the singer's performance, and it happened with my entire record collection. I later bought and sold another pair of arguably excellent speakers (ESS Heil AMTs - I forget the model but they were 2-way bookshelves with an AMT and a 12" woofer)) due to another odd midrange issue that spoiled the connection to vocalists as well. The center image on the ESS's had a strong tendency to move around based on frequency content (or something) and was extremely distracting to me. Similarly to the Advents though, the ESS's passed "critical evaluations" beautifully, and it wasn't until I began to "just listen" that the problem became apparent. I should add that I sold the Advents to my dad, who liked them just fine (and in the much less critical, usually off-axis living room listening environment, so did I), and I sold the ESS's to another engineer (we were both at The Hit Factory at the time), and he liked them fine too. I guess you could say that I'm a stickler for a specific kind of midrange transparency, and it isn't always obvious in initial "critical listening" how it will manifest itself later when enjoying music in a more intuitive state. I need to be able to "feel" the singer's...heart...soul...something...come through vividly, undisturbed, and I've found that some otherwise excellent speakers don't accomplish that very well. Most good ones do though, and I'm perfectly ok with my Genelec 1030a/1092s, JBL L-100s (what I finally replaced the KLHs with, and ultimately kept, in 1977, and still use in my living room, mostly for movies, as fronts/LF for a 5.1 system), and even the otherwise pretty annoying NS10ms I still use in the control room for mix-check. Ted Spencer, NYC David Satz wrote: Ted, one thing you might want to consider is that in the 1970s, you presumably bought a phono cartridge that you liked the sound of when it was used with your KLH speakers. Change the speakers, and that choice of cartridge is no longer the same choice you would most likely have made. The large Advents had a more forward sound than any KLHs that I can recall from that period. A more insidious hidden issue is your selection of recordings that served as the touchstone for your emotional responses. Your KLH speakers (with your existing phono cartridge) let you enjoy recordings that, through less rolled-off sounding speakers, would have sounded too bright or perhaps too closely miked. That's probably good--most record producers like "hotter" sound than most record listeners, especially listeners who attend classical concerts and have any idea what real music sounds like in a hall. But by the time you've spent six months or a year with the KLH-based system, your roster of favorite recordings has been influenced by the combined sonic characteristics of your particular cartridge and loudspeakers. If you had started off with the Advents but used the same phono cartridge as with the KLHs, then assuming that you're open to a wide enough range of musical repertoire, you might have been more impressed with flashier recordings that the Advents were effective at showing off. For example, if you enjoyed listening to string quartets with the KLHs, the Advents might have had you humming along to Mahler symphonies or "The Planets." _Then_ if you switched to KLHs, you would miss the color and presence of the Advents, with very little to compensate for their loss, since the effectiveness of those recordings depends so much on that presence and color. A mellow presentation of dramatic large-scale pieces is a different taste that you might need to acquire. Do you see what I'm getting at here? --best regards |
#37
|
|||
|
|||
In article ,
"William Sommerwerck" wrote: What is needed -- and I could name several well-known people who agree with me -- is long-term blind listening tests in which people simply sit down and listen for pleasure. Properly conducted, such testing would would provide useful information about "how" people listen, what they think they hear, and establish a baseline for judging "subjective" and "objective" testing. But such testing would require many listeners, take a lot of time, and be difficult to implement and run correctly. Not to mention the fact that both subjectivists and objectivists have a vested interest in believing what they want to believe. People are uncomfortable changing their world views. How about an AB type switch box with a 1 day timer that flips randomly at 3AM every day? Sit down and listen all day, then log what you think. Afterwards, look for a statistical correlation between the listener's log and the AB switcher's log. |
#38
|
|||
|
|||
On Sat, 14 May 2005 03:35:34 GMT, Ralph Barone
wrote: How about an AB type switch box with a 1 day timer that flips randomly at 3AM every day? Sit down and listen all day, then log what you think. Afterwards, look for a statistical correlation between the listener's log and the AB switcher's log. This sounds really good to me. It's blind, the time frame is probably spot on, and the log will hopefully be both non-threatening and have a positive (will increase in usefulness/ validity with practice) learning curve. Great! And for those who listen exclusively to a hard drive- convertable source, it might even be a simple adaptation of Arny's existing setup. For others, the switching hardware isn't impossible, and the software is trivial for folks who do software. Definitely do-able in 1981 Apple II Basic, for example. Chris Hornbeck |
#39
|
|||
|
|||
"Ralph Barone" wrote in message ... In article , "William Sommerwerck" wrote: What is needed -- and I could name several well-known people who agree with me -- is long-term blind listening tests in which people simply sit down and listen for pleasure. Properly conducted, such testing would would provide useful information about "how" people listen, what they think they hear, and establish a baseline for judging "subjective" and "objective" testing. But such testing would require many listeners, take a lot of time, and be difficult to implement and run correctly. Not to mention the fact that both subjectivists and objectivists have a vested interest in believing what they want to believe. People are uncomfortable changing their world views. How about an AB type switch box with a 1 day timer that flips randomly at 3AM every day? Sit down and listen all day, then log what you think. Afterwards, look for a statistical correlation between the listener's log and the AB switcher's log. Not a bad idea at all. Assuming the date and time were automatically recorded as well as the random item chosen, and that the "log" could not be tampered with after the fact. And that both items under test were always on. Let's see: $500 - !000 to put on the market by somebody already in the computerized equipment business? |
#40
|
|||
|
|||
I might even sell them some $2,500 power cables for them to plug into
the Romex cable feeding their power outlets. LOL It is laughable isn't it? The wire from the lcoal atation to the outlet is likely worth less than those crazy poewer cable (min xformers etc). "Look my Kia goes faster when I paint it Ferrari red!" |
Reply |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Forum | |||
Magazine Statitistics | Audio Opinions | |||
Using DJ Amplifiers in Home Theater | Audio Opinions | |||
Google Proof of An Unprovoked Personal Attack from Krueger | Audio Opinions |