Home |
Search |
Today's Posts |
#1
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
Just wondering if anyone's interested in discussing the validity of
blind testing; that is, the epistemological underpinning. My opinion is that valid blind testing is very difficult and no one's done it yet or even has a good idea how to do it. Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Yet conventional blind testing (ABX with many quick trials for example) does not control them. Conventional tests assume that the ear/ brain works like measuring equipment. It assumes (1) we experience the same thing each time we listen, and (2) we can perceive details in sound independently of our intentions or state of mind. But these things are not true. -Mike |
#2
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
Mike wrote:
Just wondering if anyone's interested in discussing the validity of blind testing; that is, the epistemological underpinning. My opinion is that valid blind testing is very difficult and no one's done it yet or even has a good idea how to do it. Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Yet conventional blind testing (ABX with many quick trials for example) does not control them. Conventional tests assume that the ear/ brain works like measuring equipment. It assumes (1) we experience the same thing each time we listen, and (2) we can perceive details in sound independently of our intentions or state of mind. But these things are not true. For something that doesn't work, it's odd how well DBT's worked in the development and improvement of lossy codecs. Not to mention Toole/Olive's work on speakers. Not to mention experimental perceptual psychology. Btw, your assumptions are off. in 'conventional ABX', you aren't limited to listening once. You're supposed to listen until you can confidently make a call that X is either A or B. And no one says you can't repeat the test. ___ -S "As human beings, we understand the world through simile, analogy, metaphor, narrative and, sometimes, claymation." - B. Mason |
#3
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Jul 15, 8:18 pm, Mike wrote:
Just wondering if anyone's interested in discussing the validity of blind testing; that is, the epistemological underpinning. Not with someone who doesn't know what epistemology is. "The validity of blind testing" is not a philosophical issue, unless you want to debate the validity of the scientific method in general. Blind tests, like all scientific tests, are valid to the extent that they give us reliable, repeatable results that are not inconsistent with other data. Now, if you can cite data that conflict with blind test results, we'd have something to talk about. But I'll bet you can't. My opinion is that valid blind testing is very difficult So is quantum mechanics. But just because you can't do it at home doesn't mean it's wrong. and no one's done it yet or even has a good idea how to do it. That's quite an opinion! Have you tried it out with any professors of psychology who specialize in perception? Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Only if it's trying to be a test of how we experience music. But DBTs aren't testing that. They are testing the question, "What sounds do and do not reach our brains?" Maybe you have a theory about how sounds that do not reach our brains can nonetheless affect how we experience music. If so, I'm not sure I want to hear it. Yet conventional blind testing (ABX with many quick trials for example) does not control them. Conventional tests assume that the ear/ brain works like measuring equipment. It assumes (1) we experience the same thing each time we listen, No, they don't at all. In fact, they control for this by requiring mulitple trials and/or subjects. and (2) we can perceive details in sound independently of our intentions or state of mind. They don't assume this either. In fact, those conducting good DBTs go to some lengths to make sure that their subjects are in a state of mind that is conducive to detecting minute sonic differences.Unlike you, however, these experts actually know something about that state of mind. bob |
#4
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
"Mike" wrote in message
... Just wondering if anyone's interested in discussing the validity of blind testing; that is, the epistemological underpinning. My opinion is that valid blind testing is very difficult and no one's done it yet or even has a good idea how to do it. Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Yet conventional blind testing (ABX with many quick trials for example) does not control them. Conventional tests assume that the ear/ brain works like measuring equipment. It assumes (1) we experience the same thing each time we listen, and (2) we can perceive details in sound independently of our intentions or state of mind. But these things are not true. I think difference between first and second hearing (for instance) is a substantial factor in audio tests. |
#5
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Sun, 15 Jul 2007 17:18:52 -0700, Mike wrote
(in article ): Just wondering if anyone's interested in discussing the validity of blind testing; that is, the epistemological underpinning. My opinion is that valid blind testing is very difficult and no one's done it yet or even has a good idea how to do it. Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Yet conventional blind testing (ABX with many quick trials for example) does not control them. Conventional tests assume that the ear/ brain works like measuring equipment. It assumes (1) we experience the same thing each time we listen, and (2) we can perceive details in sound independently of our intentions or state of mind. But these things are not true. -Mike I dunno about the epistemological underpinnings. But I have been on many double-blind ABX listening panels and have set up more than a few myself. I always liked to use audiophile and non-audiophile friends, and get a non-interested third party (read that: wife/girlfriend) to do the switching after volume is carefully calibrated using a trusty Radio-Shack (or equivalent) sound level meter. The results were never promising in my experience. If the volumes were carefully matched, then little or no results were obtained, and if not, the louder of the components under test, always was picked as the best. In my experience, the human perception of music is too complex to yield to ABX differences and that differences that I couldn't hear in direct comparison were easily heard on extended listening sessions with each component (again after carefully matching loudness using a SPL meter). |
#6
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Jul 15, 9:07 pm, "MC" wrote:
"Mike" wrote in message ... Just wondering if anyone's interested in discussing the validity of blind testing; that is, the epistemological underpinning. My opinion is that valid blind testing is very difficult and no one's done it yet or even has a good idea how to do it. Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Yet conventional blind testing (ABX with many quick trials for example) does not control them. Conventional tests assume that the ear/ brain works like measuring equipment. It assumes (1) we experience the same thing each time we listen, and (2) we can perceive details in sound independently of our intentions or state of mind. But these things are not true. I think difference between first and second hearing (for instance) is a substantial factor in audio tests. I agree. The experience of music changes from hearing to hearing. Composers exploit this when they choose to repeat material sometimes, and not others. A conventional blind test requires that subjects have the same experience each time they listen. That may be dozens of times! That's a huge effect relative to how people experience music. Mike |
#7
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Jul 15, 9:04 pm, Steven Sullivan wrote:
Mike wrote: Just wondering if anyone's interested in discussing the validity of blind testing; that is, the epistemological underpinning. My opinion is that valid blind testing is very difficult and no one's done it yet or even has a good idea how to do it. Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Yet conventional blind testing (ABX with many quick trials for example) does not control them. Conventional tests assume that the ear/ brain works like measuring equipment. It assumes (1) we experience the same thing each time we listen, and (2) we can perceive details in sound independently of our intentions or state of mind. But these things are not true. For something that doesn't work, it's odd how well DBT's worked in the development and improvement of lossy codecs. Not to mention Toole/Olive's work on speakers. Not to mention experimental perceptual psychology. A framework produces results which are valid within that framework. Producing consistent results is not surprising. The question is whether conventional DBT's provide data on the experience of music, and how that experience is affected by equipment. I don't object to "blindness", but I question the validity of most tests. Btw, your assumptions are off. in 'conventional ABX', you aren't limited to listening once. You're supposed to listen until you can confidently make a call that X is either A or B. And no one says you can't repeat the test. I didn't assume anything about how many times you listen. Mike |
#8
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Jul 15, 9:06 pm, bob wrote:
On Jul 15, 8:18 pm, Mike wrote: Just wondering if anyone's interested in discussing the validity of blind testing; that is, the epistemological underpinning. Not with someone who doesn't know what epistemology is. Would you care to tell me what your definition is, rather than making snipes? Blind tests, like all scientific tests, are valid to the extent that they give us reliable, repeatable results that are not inconsistent with other data. Now, if you can cite data that conflict with blind test results, we'd have something to talk about. But I'll bet you can't. I'm suggesting that you don't have the data I'm interested in. You have a lot of data on what happens when you treat ears and brains like measurement instruments. You have no data on how ears and brains behave under other conditions. Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Only if it's trying to be a test of how we experience music. But DBTs aren't testing that. They are testing the question, "What sounds do and do not reach our brains?" I think even you would disagree with that definition if you thought about it. They are testing whether two different sounds can be told apart, or to say it another way, whether they produce two different experiences. You can't get away from the experience of music unless you are trying to arbitrarily separate the experience of sound from the experience of music. That's one poor assumption. Maybe you have a theory about how sounds that do not reach our brains can nonetheless affect how we experience music. If so, I'm not sure I want to hear it. Yet conventional blind testing (ABX with many quick trials for example) does not control them. Conventional tests assume that the ear/ brain works like measuring equipment. It assumes (1) we experience the same thing each time we listen, No, they don't at all. In fact, they control for this by requiring mulitple trials and/or subjects. Exactly.. they require multiple trials while not controlling what's going on inside your brain during each trial. and (2) we can perceive details in sound independently of our intentions or state of mind. They don't assume this either. In fact, those conducting good DBTs go to some lengths to make sure that their subjects are in a state of mind that is conducive to detecting minute sonic differences. Right.. they have defined "minute sonic differences" as something different than perceiving music.That definition itself becomes a limiting factor in how well you can generalize the results. Unlike you, however, these experts actually know something about that state of mind. Are you interested in discussion or launching insults? Mike |
#9
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
George Graves wrote:
On Sun, 15 Jul 2007 17:18:52 -0700, Mike wrote (in article ): Just wondering if anyone's interested in discussing the validity of blind testing; that is, the epistemological underpinning. My opinion is that valid blind testing is very difficult and no one's done it yet or even has a good idea how to do it. Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Yet conventional blind testing (ABX with many quick trials for example) does not control them. Conventional tests assume that the ear/ brain works like measuring equipment. It assumes (1) we experience the same thing each time we listen, and (2) we can perceive details in sound independently of our intentions or state of mind. But these things are not true. -Mike I dunno about the epistemological underpinnings. But I have been on many double-blind ABX listening panels and have set up more than a few myself. I always liked to use audiophile and non-audiophile friends, and get a non-interested third party (read that: wife/girlfriend) to do the switching after volume is carefully calibrated using a trusty Radio-Shack (or equivalent) sound level meter. The results were never promising in my experience. If the volumes were carefully matched, then little or no results were obtained, and if not, the louder of the components under test, always was picked as the best. Why are such resuls 'not promising'? In my experience, the human perception of music is too complex to yield to ABX differences is your experience simply anecdotal or has it got any rigorous backing? and that differences that I couldn't hear in direct comparison were easily heard on extended listening sessions with each component (again after carefully matching loudness using a SPL meter). And, at *that* point, why couldn't you validate the differences with ABX? It's always puzzling to me why the 'you need extended listening sessions' faction doesn't perform the obvious 'slam dunk' experiment. If it takes extended listening to develop a confident perception of difference, fine: there's no intrinsic 'prohibition' against that in ABX testing. All that's *non-negotiable* is that, when it comes time to actually demonstrate the ability to tell the difference, that the choice be made 'blind'. ___ -S "As human beings, we understand the world through simile, analogy, metaphor, narrative and, sometimes, claymation." - B. Mason |
#10
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On 16 Jul 2007 04:08:05 GMT, George Graves wrote:
The results were never promising in my experience. If the volumes were carefully matched, then little or no results were obtained IOW, the audible differences audiophiles claim to hear don't actually exist. Isn't that great? You can find something more productive to waste your money on. ABX differences and that differences that I couldn't hear in direct comparison were easily heard on extended listening sessions Yeah. Non-blind listening sessions, I bet :-) |
#11
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
George Graves wrote:
... If the volumes were carefully matched, then little or no results were obtained, and if not, the louder of the components under test, always was picked as the best. Hold on there. The point of a double-blind test is to determine whether the subject can hear a *difference* between the samples. Not to determine which one is "best" (whatever that means). If the volume is not matched, then of course the subject can hear a difference - "sample A is louder". //Walt |
#12
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
"Mike" wrote in message
Just wondering if anyone's interested in discussing the validity of blind testing; that is, the epistemological underpinning. Epistemology is the study of why we believe what we believe, not what we believe. My opinion is that valid blind testing is very difficult and no one's done it yet or even has a good idea how to do it. This seems to be more like a discussion of what you believe, than a discussion of why you believe it. Are you sure that you know what an epistemological discussion would be like? Personally, I base as many of my beliefs as I can on my personal experiences, particularly those experiences which are as unaffected by my previous state of mind as possible. IOW, I try to have my beliefs about sonmething rise above my specific prejudices about that thing. One consequence of this desire is that I prefer to base my beliefs on bias-controlled tests wherever possible. Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. So are you saying you think that we should be open to perceiving that which is there to be perceived? Yet conventional blind testing (ABX with many quick trials for example) does not control them. Seems like a straw man argument since ABX tests need not involve many quick trials. Many ABX tests have been done with a few extended trials, for example. Conventional tests assume that the ear/ brain works like measuring equipment. It's well known that human listening is often very inconsistent, and can be affected by the listener's beliefs. I know of no test equipment that works that way. Test equipment tends to be very consistent, and being inanimate, it has no beliefs at all to bias it. |
#13
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Jul 16, 6:54 pm, Mike wrote:
On Jul 15, 9:06 pm, bob wrote: Blind tests, like all scientific tests, are valid to the extent that they give us reliable, repeatable results that are not inconsistent with other data. Now, if you can cite data that conflict with blind test results, we'd have something to talk about. But I'll bet you can't. I'm suggesting that you don't have the data I'm interested in. So you just ignore it? That's hardly the appropriate posture for someone who claims to want a discussion. And at least I HAVE data. You have a lot of data on what happens when you treat ears and brains like measurement instruments. You have no data on how ears and brains behave under other conditions. You are playing semantic games here. ABX subjects are using their ears to *hear.* If you think that hearing operates differently under different conditions, then it is incumbent on you to a) define what those conditions are, and b) provide evidence that it does, indeed, operate differently. You haven't--and cannot--do that. So what's to discuss? Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Only if it's trying to be a test of how we experience music. But DBTs aren't testing that. They are testing the question, "What sounds do and do not reach our brains?" I think even you would disagree with that definition if you thought about it. They are testing whether two different sounds can be told apart, Which is another way of stating what I said. or to say it another way, whether they produce two different experiences. You can't get away from the experience of music unless you are trying to arbitrarily separate the experience of sound from the experience of music. That's one poor assumption. I'm not "getting away from the experience of music." You're the one making the claims--and the wild assumptions--here. You're assuming, or claiming, that "experiencing music" will somehow change the sensitivity of our hearing. But you have no evidence for this. Indeed, available evidence suggests the opposite--that extended music listening is a LESS sensitive way to test for audible differences. bob |
#14
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Mon, 16 Jul 2007 15:52:52 -0700, ScottW wrote
(in article ): "George Graves" wrote in message ... On Sun, 15 Jul 2007 17:18:52 -0700, Mike wrote (in article ): Just wondering if anyone's interested in discussing the validity of blind testing; that is, the epistemological underpinning. My opinion is that valid blind testing is very difficult and no one's done it yet or even has a good idea how to do it. Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Yet conventional blind testing (ABX with many quick trials for example) does not control them. Conventional tests assume that the ear/ brain works like measuring equipment. It assumes (1) we experience the same thing each time we listen, and (2) we can perceive details in sound independently of our intentions or state of mind. But these things are not true. -Mike I dunno about the epistemological underpinnings. But I have been on many double-blind ABX listening panels and have set up more than a few myself. I always liked to use audiophile and non-audiophile friends, and get a non-interested third party (read that: wife/girlfriend) to do the switching after volume is carefully calibrated using a trusty Radio-Shack (or equivalent) sound level meter. The results were never promising in my experience. If the volumes were carefully matched, then little or no results were obtained, and if not, the louder of the components under test, always was picked as the best. In my experience, the human perception of music is too complex to yield to ABX differences and that differences that I couldn't hear in direct comparison were easily heard on extended listening sessions Is there a reason ABX couldn't include extended listening sessions? ScottW No, it's just that I would expect that one would get a better idea of the true sound of the DUT without the associated switching apparatus which accompanies ABX testing. |
#15
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Mon, 16 Jul 2007 15:55:20 -0700, Steven Sullivan wrote
(in article ): George Graves wrote: On Sun, 15 Jul 2007 17:18:52 -0700, Mike wrote (in article ): Just wondering if anyone's interested in discussing the validity of blind testing; that is, the epistemological underpinning. My opinion is that valid blind testing is very difficult and no one's done it yet or even has a good idea how to do it. Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Yet conventional blind testing (ABX with many quick trials for example) does not control them. Conventional tests assume that the ear/ brain works like measuring equipment. It assumes (1) we experience the same thing each time we listen, and (2) we can perceive details in sound independently of our intentions or state of mind. But these things are not true. -Mike I dunno about the epistemological underpinnings. But I have been on many double-blind ABX listening panels and have set up more than a few myself. I always liked to use audiophile and non-audiophile friends, and get a non-interested third party (read that: wife/girlfriend) to do the switching after volume is carefully calibrated using a trusty Radio-Shack (or equivalent) sound level meter. The results were never promising in my experience. If the volumes were carefully matched, then little or no results were obtained, and if not, the louder of the components under test, always was picked as the best. Why are such resuls 'not promising'? In my experience, the human perception of music is too complex to yield to ABX differences is your experience simply anecdotal or has it got any rigorous backing? and that differences that I couldn't hear in direct comparison were easily heard on extended listening sessions with each component (again after carefully matching loudness using a SPL meter). And, at *that* point, why couldn't you validate the differences with ABX? It's always puzzling to me why the 'you need extended listening sessions' faction doesn't perform the obvious 'slam dunk' experiment. If it takes extended listening to develop a confident perception of difference, fine: there's no intrinsic 'prohibition' against that in ABX testing. All that's *non-negotiable* is that, when it comes time to actually demonstrate the ability to tell the difference, that the choice be made 'blind'. I'm sorry. I simply cannot answer your questions. I'm merely going by my own experience as a listener. And this argument about subjective vs objective testing methodologies have been going on much longer than we've been discussing it, and nobody has ever, to my knowledge been able resolve the argument. I used believe explicitly in objective ABX testing until I became absolutely convinced that it really wasn't very good at characterizing the differences that one hears between components. |
#16
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Mon, 16 Jul 2007 15:56:01 -0700, Michael Warner wrote
(in article ): On 16 Jul 2007 04:08:05 GMT, George Graves wrote: The results were never promising in my experience. If the volumes were carefully matched, then little or no results were obtained IOW, the audible differences audiophiles claim to hear don't actually exist. Isn't that great? You can find something more productive to waste your money on. ABX differences and that differences that I couldn't hear in direct comparison were easily heard on extended listening sessions Yeah. Non-blind listening sessions, I bet :-) And it could be otherwise, how? I'm the one replacing my reference component with the DUT. Of course its non-blind. |
#17
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
"Walt" wrote in message
... George Graves wrote: ... If the volumes were carefully matched, then little or no results were obtained, and if not, the louder of the components under test, always was picked as the best. Hold on there. The point of a double-blind test is to determine whether the subject can hear a *difference* between the samples. Not to determine which one is "best" (whatever that means). If the volume is not matched, then of course the subject can hear a difference - "sample A is louder". //Walt No, that is the point of an ABX test. It is perfectly scientific to have a blind A-B preference test. Blindness is a test condition applicable to many different types of tests. |
#18
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Jul 16, 12:08 am, George Graves wrote:
In my experience, the human perception of music is too complex to yield to ABX differences and that differences that I couldn't hear in direct comparison were easily heard on extended listening sessions with each component (again after carefully matching loudness using a SPL meter). In other words, you changed not one but two conditions of your comparison: 1) how long you listened, and 2) how much visual information you had Then you assumed that the differing result came because of #1. The fallacy of this reasoning should be obvious. bob |
#19
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
Mike wrote:
On Jul 15, 9:07 pm, "MC" wrote: "Mike" wrote in message ... Just wondering if anyone's interested in discussing the validity of blind testing; that is, the epistemological underpinning. My opinion is that valid blind testing is very difficult and no one's done it yet or even has a good idea how to do it. Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Yet conventional blind testing (ABX with many quick trials for example) does not control them. Conventional tests assume that the ear/ brain works like measuring equipment. It assumes (1) we experience the same thing each time we listen, and (2) we can perceive details in sound independently of our intentions or state of mind. But these things are not true. I think difference between first and second hearing (for instance) is a substantial factor in audio tests. I agree. The experience of music changes from hearing to hearing. Composers exploit this when they choose to repeat material sometimes, and not others. A conventional blind test requires that subjects have the same experience each time they listen. That may be dozens of times! That's a huge effect relative to how people experience music. These analogies keep getting sillier and sillier. A 'conventional blind test' requires nothing of the sort. It requires that the subject be 'blinded' to the source of the sound -- and thus use only their ears, to judge what they're hearing. That's all, essentially. Every recommedation beyond that is geared to *increasing* the subject's chances of discerning an audible difference. Instead of hypothesizing bogus 'problems' with blind testing, how about acknowledging the intrinsic, inescapable, well-established issues accompanying ANY 'sighted' evaluation of audio? ___ -S "As human beings, we understand the world through simile, analogy, metaphor, narrative and, sometimes, claymation." - B. Mason |
#20
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
Mike wrote:
and do not reach our brains?" I think even you would disagree with that definition if you thought about it. They are testing whether two different sounds can be told apart, or to say it another way, whether they produce two different experiences. You can't get away from the experience of music unless you are trying to arbitrarily separate the experience of sound from the experience of music. That's one poor assumption. I can listen to the eact same recorded performance of a work on the exact same rig -- even minutes apart -- and have different 'experiences' of it. For example, I might be more bored by it the second time. But of course the *sound* hasn't changed at all. Yet by your logic, I've listened to two different sounds. Imagine if that logic were carried over into all scientific work. Maybe you have a theory about how sounds that do not reach our brains can nonetheless affect how we experience music. If so, I'm not sure I want to hear it. Yet conventional blind testing (ABX with many quick trials for example) does not control them. Conventional tests assume that the ear/ brain works like measuring equipment. It assumes (1) we experience the same thing each time we listen, No, they don't at all. In fact, they control for this by requiring mulitple trials and/or subjects. Exactly.. they require multiple trials while not controlling what's going on inside your brain during each trial. That's the POINT. Your *brain* will quite readily 'hear' difference where *none exists*. The old 'phantom switch' setup is a classic example, where literally nothing is changed, but the listener is led to believe that something has been changed. Usually they report hearing 'differnece' -- sometimes a comically large one. So, let the brain 'do' what it likes. Running the comparison 'blind' takes care of its tendency to overestimate the occurence of 'difference'. Meanwhile, with training, some tiny measurable differences -- e.g., on the order of 0.2 dB in some frequency ranges -- can be confidently shown to be discernable via DBT. Others...not so much. Which makes sense; no one should expect that all measurable differences are perceptible (whereas all perceptible differences so far, have turned out to have a measurement affected by them, indirectly if not directly). and (2) we can perceive details in sound independently of our intentions or state of mind. They don't assume this either. In fact, those conducting good DBTs go to some lengths to make sure that their subjects are in a state of mind that is conducive to detecting minute sonic differences. Right.. they have defined "minute sonic differences" as something different than perceiving music.That definition itself becomes a limiting factor in how well you can generalize the results. "perceived" difference in music, as for any sound, can occur even when the music itself is exactly the same. How do you propose to get around this fundamental problem? ___ -S "As human beings, we understand the world through simile, analogy, metaphor, narrative and, sometimes, claymation." - B. Mason |
#21
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
Walt wrote:
George Graves wrote: ... If the volumes were carefully matched, then little or no results were obtained, and if not, the louder of the components under test, always was picked as the best. Hold on there. The point of a double-blind test is to determine whether the subject can hear a *difference* between the samples. Not to determine which one is "best" (whatever that means). DBT can be used to determine if preference is due to sound, or to something else. Of course, it will already have been determined that the two things sound *different*. Such is the case for using DBTs to study loudspeaker preference. ___ -S "As human beings, we understand the world through simile, analogy, metaphor, narrative and, sometimes, claymation." - B. Mason |
#22
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
George Graves wrote:
On Mon, 16 Jul 2007 15:52:52 -0700, ScottW wrote (in article ): "George Graves" wrote in message ... On Sun, 15 Jul 2007 17:18:52 -0700, Mike wrote (in article ): Just wondering if anyone's interested in discussing the validity of blind testing; that is, the epistemological underpinning. My opinion is that valid blind testing is very difficult and no one's done it yet or even has a good idea how to do it. Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Yet conventional blind testing (ABX with many quick trials for example) does not control them. Conventional tests assume that the ear/ brain works like measuring equipment. It assumes (1) we experience the same thing each time we listen, and (2) we can perceive details in sound independently of our intentions or state of mind. But these things are not true. -Mike I dunno about the epistemological underpinnings. But I have been on many double-blind ABX listening panels and have set up more than a few myself. I always liked to use audiophile and non-audiophile friends, and get a non-interested third party (read that: wife/girlfriend) to do the switching after volume is carefully calibrated using a trusty Radio-Shack (or equivalent) sound level meter. The results were never promising in my experience. If the volumes were carefully matched, then little or no results were obtained, and if not, the louder of the components under test, always was picked as the best. In my experience, the human perception of music is too complex to yield to ABX differences and that differences that I couldn't hear in direct comparison were easily heard on extended listening sessions Is there a reason ABX couldn't include extended listening sessions? ScottW No, it's just that I would expect that one would get a better idea of the true sound of the DUT without the associated switching apparatus which accompanies ABX testing. That's not necessarily a well-warranted expectation -- the 'switch' can range from manually changing the source, to using a sophisticated device like the ABX Comparator -- but by all means, if you think a switch interfered with 'getting a better idea of the true sound', then listen *without the switch* until you are sure you can identify the true sound -- *then* do the ABX! ___ -S "As human beings, we understand the world through simile, analogy, metaphor, narrative and, sometimes, claymation." - B. Mason |
#23
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
George Graves wrote:
On Mon, 16 Jul 2007 15:55:20 -0700, Steven Sullivan wrote faction doesn't perform the obvious 'slam dunk' experiment. If it takes extended listening to develop a confident perception of difference, fine: there's no intrinsic 'prohibition' against that in ABX testing. All that's *non-negotiable* is that, when it comes time to actually demonstrate the ability to tell the difference, that the choice be made 'blind'. I'm sorry. I simply cannot answer your questions. I'm merely going by my own experience as a listener. And this argument about subjective vs objective testing methodologies have been going on much longer than we've been discussing it, and nobody has ever, to my knowledge been able resolve the argument. I used believe explicitly in objective ABX testing until I became absolutely convinced that it really wasn't very good at characterizing the differences that one hears between components. It sounds like you ahve 'resolved the argument' in your own mind -- based on *what*? A 'slam-dunk' experiment as I described, or you have one of your own? Or was it just, again, sighted comparison -- a 'protocol' whose flaws are not seriously questioned by researchers into perception...or even orchestra audition panels. ___ -S "As human beings, we understand the world through simile, analogy, metaphor, narrative and, sometimes, claymation." - B. Mason |
#24
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Mon, 16 Jul 2007 20:30:19 -0700, Harry Lavo wrote
(in article ): "Walt" wrote in message ... George Graves wrote: ... If the volumes were carefully matched, then little or no results were obtained, and if not, the louder of the components under test, always was picked as the best. Hold on there. The point of a double-blind test is to determine whether the subject can hear a *difference* between the samples. Not to determine which one is "best" (whatever that means). If the volume is not matched, then of course the subject can hear a difference - "sample A is louder". //Walt No, that is the point of an ABX test. It is perfectly scientific to have a blind A-B preference test. Blindness is a test condition applicable to many different types of tests. True. It just doesn't seem applicable to audio and the way people perceive sound. |
#25
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Mon, 16 Jul 2007 20:31:39 -0700, bob wrote
(in article ): On Jul 16, 12:08 am, George Graves wrote: In my experience, the human perception of music is too complex to yield to ABX differences and that differences that I couldn't hear in direct comparison were easily heard on extended listening sessions with each component (again after carefully matching loudness using a SPL meter). In other words, you changed not one but two conditions of your comparison: 1) how long you listened, and 2) how much visual information you had Then you assumed that the differing result came because of #1. The fallacy of this reasoning should be obvious. bob Both of your assumptions are just that. But I'm not here to argue this point. You want ABX comparisons? Fine. You pick your audio system based on the criteria you find important, and I pick my components based upon how the equipment sounds to me. What's interesting is that when we "subjective testers" get together, we all seem to find the same characteristics for the same equipment, even though we all came by these "sonic signatures" independently of one another on our own systems and they usually don't show up under ABX testing conditions. Mass hysteria? You tell me. |
#26
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
"Mike" wrote in message
I agree. The experience of music changes from hearing to hearing. I think you mean that we experience music differently every time we hear it. Composers exploit this when they choose to repeat material sometimes, and not others. OK. A conventional blind test requires that subjects have the same experience each time they listen. Wrong. In fact properly designed listening tests account for the fact that we have an at least slightly different experience every time we hear the same segment of music. It is conventional audiophile sighted listening tests that are based on the idea that subjects have the same experience each time they listen. |
#27
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
"George Graves" wrote in message
No, it's just that I would expect that one would get a better idea of the true sound of the DUT without the associated switching apparatus which accompanies ABX testing. In fact it is relatively easy to do bias-controlled including ABX listening tests with no extra switching apparatus at all. |
#28
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
"George Graves" wrote in message
I'm sorry. I simply cannot answer your questions. I'm merely going by my own experience as a listener. Does that experience include actually doing any bias-controlled listening tests? And this argument about subjective vs objective testing methodologies have been going on much longer than we've been discussing it, and nobody has ever, to my knowledge been able resolve the argument. Actually, the argument has been resolved for a long time. Anybody who wants to do a serious listening test, that is one where the results have a great deal of credibility, does their tests with appropriate bias controls in place. I used believe explicitly in objective ABX testing until I became absolutely convinced that it really wasn't very good at characterizing the differences that one hears between components. How did you become convinced that way? Based on your posts, you became convinced of a lot things that are not true along the way. |
#29
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
"George Graves" wrote in message
On Mon, 16 Jul 2007 15:56:01 -0700, Michael Warner wrote (in article ): On 16 Jul 2007 04:08:05 GMT, George Graves wrote: The results were never promising in my experience. If the volumes were carefully matched, then little or no results were obtained IOW, the audible differences audiophiles claim to hear don't actually exist. Isn't that great? You can find something more productive to waste your money on. ABX differences and that differences that I couldn't hear in direct comparison were easily heard on extended listening sessions Yeah. Non-blind listening sessions, I bet :-) I might question the level matching. And it could be otherwise, how? I'm the one replacing my reference component with the DUT. Of course its non-blind. Long term listening tests have been done where the replacement process was hidden from the listener, and randomized. |
#30
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
Arny Krueger wrote:
Epistemology is the study of *why* we believe what we believe, not what we believe. I think it would be more correct to say that epistemology is the science of the limits of knowledge, and, in a way, how we can ascertain knowledge within these limits (method). I see it more as the "what" than the "why." Speaking loosely, "why" we believe encompasses the psychology behind belief--something a bit different from epistemology proper. This also turns on a distinction between belief and knowledge. The latter implies the former, but it is not necessarily the other way around. mp |
#31
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
"Arny Krueger" wrote in message
... A conventional blind test requires that subjects have the same experience each time they listen. Wrong. In fact properly designed listening tests account for the fact that we have an at least slightly different experience every time we hear the same segment of music. Right. A well-designed double-blind test will try the experiment several times, varying the order. |
#32
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Jul 16, 4:00 pm, "Arny Krueger" wrote:
"Mike" wrote in message Conventional tests assume that the ear/ brain works like measuring equipment. It's well known that human listening is often very inconsistent, and can be affected by the listener's beliefs. I know of no test equipment that works that way. Test equipment tends to be very consistent, and being inanimate, it has no beliefs at all to bias it. Exactly, that's why it is such a poor assumption that people experience the same thing each time they listen to a section (small or large) of music. I've never heard of an ABX test that didn't assume that. Mike |
#33
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Jul 16, 8:26 pm, bob wrote:
On Jul 16, 6:54 pm, Mike wrote: On Jul 15, 9:06 pm, bob wrote: You have a lot of data on what happens when you treat ears and brains like measurement instruments. You have no data on how ears and brains behave under other conditions. You are playing semantic games here. ABX subjects are using their ears to *hear.* If you think that hearing operates differently under different conditions, then it is incumbent on you to a) define what those conditions are, and b) provide evidence that it does, indeed, operate differently. You haven't--and cannot--do that. So what's to discuss? I would like to discuss what assumptions one has to make about the ear/ brain to investigate it. You write as though this were some kind of personal battle of opposing viewpoints. You also seem to need things to be precisely defined in order to believe they exist. It's obvious hearing operates differently under different conditions. For example, you have many choices about _what_ to listen for, and you can easily choose one thing but miss something else, so your intention is one condition. But every blind test I'm aware of makes the assumption that we have the same experience each time we listen to something. That's a very poor assumption, and that makes the results suspect. If it's incumbent on anyone to provide proof, it's incumbent on those who do the experiments to show that they have a firm basis. Since how we listen, what we listen for, what we're open to perceiving (and so on) affect our experience of music, then a valid test would control these things. Only if it's trying to be a test of how we experience music. But DBTs aren't testing that. They are testing the question, "What sounds do and do not reach our brains?" I think even you would disagree with that definition if you thought about it. They are testing whether two different sounds can be told apart, Which is another way of stating what I said. Your statement is very loose, though. It implies we are testing whether a signal is "large enough" to reach our brains, which we are not testing. Both A and B reach our brains. or to say it another way, whether they produce two different experiences. You can't get away from the experience of music unless you are trying to arbitrarily separate the experience of sound from the experience of music. That's one poor assumption. I'm not "getting away from the experience of music." You're the one making the claims--and the wild assumptions--here. You're assuming, or claiming, that "experiencing music" will somehow change the sensitivity of our hearing. But you have no evidence for this. You're assuming it does not change the sensitivity, and provide no evidence for that. Indeed, available evidence suggests the opposite--that extended music listening is a LESS sensitive way to test for audible differences. "Extended" vs "non-extended" is not the point. Even "enjoying music" is not the point. The point is that our experience as a whole changes each time we listen. Mike |
#34
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Tue, 17 Jul 2007 15:55:51 -0700, Steven Sullivan wrote
(in article ): George Graves wrote: On Mon, 16 Jul 2007 15:55:20 -0700, Steven Sullivan wrote faction doesn't perform the obvious 'slam dunk' experiment. If it takes extended listening to develop a confident perception of difference, fine: there's no intrinsic 'prohibition' against that in ABX testing. All that's *non-negotiable* is that, when it comes time to actually demonstrate the ability to tell the difference, that the choice be made 'blind'. I'm sorry. I simply cannot answer your questions. I'm merely going by my own experience as a listener. And this argument about subjective vs objective testing methodologies have been going on much longer than we've been discussing it, and nobody has ever, to my knowledge been able resolve the argument. I used believe explicitly in objective ABX testing until I became absolutely convinced that it really wasn't very good at characterizing the differences that one hears between components. It sounds like you ahve 'resolved the argument' in your own mind -- based on *what*? A 'slam-dunk' experiment as I described, or you have one of your own? Or was it just, again, sighted comparison -- a 'protocol' whose flaws are not seriously questioned by researchers into perception...or even orchestra audition panels. What difference does it make? I'm not trying to convince you (or anyone else) of anything. I merely stated what my 40 years of being an audio enthusiast and 30 years of writing about it have told me. You don't accept it? That's OK. It's up to you to come to your own conclusions, and in the final analysis, what anyone else finds to be true on such a subjective issue as sound is totally irrelevant. |
#35
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Tue, 17 Jul 2007 16:02:07 -0700, Arny Krueger wrote
(in article ): "George Graves" wrote in message On Mon, 16 Jul 2007 15:56:01 -0700, Michael Warner wrote (in article ): On 16 Jul 2007 04:08:05 GMT, George Graves wrote: The results were never promising in my experience. If the volumes were carefully matched, then little or no results were obtained IOW, the audible differences audiophiles claim to hear don't actually exist. Isn't that great? You can find something more productive to waste your money on. ABX differences and that differences that I couldn't hear in direct comparison were easily heard on extended listening sessions Yeah. Non-blind listening sessions, I bet :-) I might question the level matching. And it could be otherwise, how? I'm the one replacing my reference component with the DUT. Of course its non-blind. Long term listening tests have been done where the replacement process was hidden from the listener, and randomized. Yeah, I've done them too. It works. I can listen to a component where I don't know what I'm listening too and make comments that characterize the sound I hear. But what of it? Am I commenting on one component or on the entire playback chain. If I'm not intimately familiar with the sound of the entire system, then it's obvious that my findings are about the latter. Not too useful. But if someone comes into my home, and replaces a component blindly so that I don't know what it is, and I use material with which I'm familiar with which to listen, then any changes from what I normally hear are obviously going to be attributable to whatever component was exchanged (or the way in which the "foreign" component interacts with the rest of my system). Wouldn't this make sense? |
#36
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
"George Graves" wrote in message
What's interesting is that when we "subjective testers" get together, we all seem to find the same characteristics for the same equipment, even though we all came by these "sonic signatures" independently of one another on our own systems. True, that subjective reviewers tend to draw their descriptive words and phrases from a relatively small pool, and there's a tendency for subjective reviews to all sound the same. |
#37
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
"Walt" wrote in message
George Graves wrote: ... If the volumes were carefully matched, then little or no results were obtained, and if not, the louder of the components under test, always was picked as the best. Hold on there. The point of a double-blind test is to determine whether the subject can hear a *difference* between the samples. Not necessarily. The point of a double-blind test is to control the parameters that are apparent to the listener. The basic test can be a wide variety of tests, including preference testing. One of the most common kind of double blind listening tests is called the ABC/hr test which is a test of perceived level of degradation. While not exactly a preference test, and while it can yield information about the listener's ability to hear differences, it is akin to a preference test, presuming that listeners prefer the original sound. Not to determine which one is "best" (whatever that means). If there is an agreement about audiophiles and professionals about what sounds best, the most universal stated preference is for uncolored sound that is as much as like the original sound. If the volume is not matched, then of course the subject can hear a difference - "sample A is louder". No doubt. All listening tests should be in some sense level-matched. However, if the equipment being compared has significant colorations, then the true meaning of level-matching becomes less clear. I'll bet money Walt that you have never before heard of ABC/hr listening tests, just like you obviously never ever before heard about long-term ABX tests. This means that you might feel the need to study up some more before making any more global pronouncements. |
#38
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
George Graves wrote:
On Mon, 16 Jul 2007 20:31:39 -0700, bob wrote On Jul 16, 12:08 am, George Graves wrote: In my experience, the human perception of music is too complex to yield to ABX differences and that differences that I couldn't hear in direct comparison were easily heard on extended listening sessions with each component (again after carefully matching loudness using a SPL meter). In other words, you changed not one but two conditions of your comparison: 1) how long you listened, and 2) how much visual information you had Then you assumed that the differing result came because of #1. The fallacy of this reasoning should be obvious. Both of your assumptions are just that. But I'm not here to argue this point. You want ABX comparisons? Fine. You pick your audio system based on the criteria you find important, and I pick my components based upon how the equipment sounds to me. What's interesting is that when we "subjective testers" get together, we all seem to find the same characteristics for the same equipment, even though we all came by these "sonic signatures" independently of one another on our own systems and they usually don't show up under ABX testing conditions. Mass hysteria? You tell me. This seems to be an example of a scientifically well-established and very common effect: See http://en.wikipedia.org/wiki/Asch_co...ty_experiments If your opinion was entirely based on your own audio perception, it would be apparent in double blind listening tests. Since it isn't, there must be some non-auditory phenomenon at work. Dr. Occam and I vote for the Asch effect. //Walt |
#39
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Wed, 18 Jul 2007 15:55:58 -0700, Arny Krueger wrote
(in article ): "George Graves" wrote in message What's interesting is that when we "subjective testers" get together, we all seem to find the same characteristics for the same equipment, even though we all came by these "sonic signatures" independently of one another on our own systems. True, that subjective reviewers tend to draw their descriptive words and phrases from a relatively small pool, and there's a tendency for subjective reviews to all sound the same. Nice rationalization, but you're overlooking something. If I say that a certain pre-amp has a sandpaper-like dryness in the upper octaves as opposed to being syruppy, or dark, or grainy, and other subjective reviewers come to the same conclusion independently, then, it would almost have to follow that this particular component, does, indeed, exhibit that characteristic, would it not? |
#40
Posted to rec.audio.high-end
|
|||
|
|||
Blind testing: the epistemology
On Wed, 18 Jul 2007 16:01:22 -0700, Walt wrote
(in article ): George Graves wrote: On Mon, 16 Jul 2007 20:31:39 -0700, bob wrote On Jul 16, 12:08 am, George Graves wrote: In my experience, the human perception of music is too complex to yield to ABX differences and that differences that I couldn't hear in direct comparison were easily heard on extended listening sessions with each component (again after carefully matching loudness using a SPL meter). In other words, you changed not one but two conditions of your comparison: 1) how long you listened, and 2) how much visual information you had Then you assumed that the differing result came because of #1. The fallacy of this reasoning should be obvious. Both of your assumptions are just that. But I'm not here to argue this point. You want ABX comparisons? Fine. You pick your audio system based on the criteria you find important, and I pick my components based upon how the equipment sounds to me. What's interesting is that when we "subjective testers" get together, we all seem to find the same characteristics for the same equipment, even though we all came by these "sonic signatures" independently of one another on our own systems and they usually don't show up under ABX testing conditions. Mass hysteria? You tell me. This seems to be an example of a scientifically well-established and very common effect: See http://en.wikipedia.org/wiki/Asch_co...ty_experiments If your opinion was entirely based on your own audio perception, it would be apparent in double blind listening tests. Since it isn't, there must be some non-auditory phenomenon at work. Dr. Occam and I vote for the Asch effect. How can the Asch effect be an issue, when another reviewer who I do not know in another publication comes to exactly the same conclusions that I have come to, when neither of us knew that the other was auditioning the same piece of equipment? I cannot speak for others, of course, but, my audio perception is based upon my memory or real, live music. I go to concerts constantly. I frequent local jazz clubs, attend concerts of symphonic as well as chamber music several times a week. While I'm sure that my memory of live music is imperfect, it makes me a better judge, I think, than someone who merely compares one audio component to another, relying on nothing for a touchstone to reality or a reference. Musical perception is extremely complex and many things can change the sound of music from one listening instance to the next, but I do know that there are certain instrumental "signatures" that manage to survive the differences imposed by venue acoustics, relative seating location, etc. and I try to cue-in on those. I also make my own recordings of many of these. I use a MiniDisc Hi-MD recorder in the 16-bit linear record mode and a Sony LT-929 'MS' stereo microphone for many of them (especially jazz club performance - and yes, I always ask permission). When I use these recordings as well as symphonic recordings that I have made using professional equipment as my source materials when evaluating equipment, I can easily tell when a piece of equipment has introduced some coloration that is not consonant with the sound of live music. None of this is perfect, to say the least, but it does make my perceptions of a piece of equipment's sonic signature similar to that of other reviewers auditioning the same make and model piece of equipment. This has happened to me so many times that I know its not coincidence. I think that John Atkinson of Stereophile posts here occasionally. I believe that if you ask him, he''ll tell you that the same thing has happened to him, probably countless times. It's not a rare occurrence. This validates long-term listening tests vs ABX tests as far as I'm concerned. |
Reply |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Forum | |||
Blind Testing - Some Further Thoughts | Audio Opinions | |||
double-blind testing | High End Audio | |||
Blind testing and audio enjoyment | Tech | |||
Comments about Blind Testing | High End Audio | |||
Equation for blind testing? | Audio Opinions |