[Mike]

Just wondering if anyone's interested in discussing the validity of
blind testing; that is, the epistemological underpinning.

My opinion is that valid blind testing is very difficult and no one's
done it yet or even has a good idea how to do it. Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.
Yet conventional blind testing (ABX with many quick trials for
example) does not control them. Conventional tests assume that the ear/
brain works like measuring equipment. It assumes (1) we experience the
same thing each time we listen, and (2) we can perceive details in
sound independently of our intentions or state of mind. But these
things are not true.

-Mike

[Steven Sullivan]

Mike wrote:
Just wondering if anyone's interested in discussing the validity of
blind testing; that is, the epistemological underpinning.

My opinion is that valid blind testing is very difficult and no one's
done it yet or even has a good idea how to do it. Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.
Yet conventional blind testing (ABX with many quick trials for
example) does not control them. Conventional tests assume that the ear/
brain works like measuring equipment. It assumes (1) we experience the
same thing each time we listen, and (2) we can perceive details in
sound independently of our intentions or state of mind. But these
things are not true.

For something that doesn't work, it's odd how well DBT's worked in the
development and improvement of lossy codecs. Not to mention Toole/Olive's
work on speakers. Not to mention experimental perceptual psychology.

Btw, your assumptions are off. in 'conventional ABX', you aren't limited
to listening once. You're supposed to listen until you can confidently
make a call that X is either A or B. And no one says you can't repeat the
test.

___
-S
"As human beings, we understand the world through simile, analogy,
metaphor, narrative and, sometimes, claymation." - B. Mason

[bob]

On Jul 15, 8:18 pm, Mike wrote:
Just wondering if anyone's interested in discussing the validity of
blind testing; that is, the epistemological underpinning.

Not with someone who doesn't know what epistemology is. "The validity
of blind testing" is not a philosophical issue, unless you want to
debate the validity of the scientific method in general.

Blind tests, like all scientific tests, are valid to the extent that
they give us reliable, repeatable results that are not inconsistent
with other data. Now, if you can cite data that conflict with blind
test results, we'd have something to talk about. But I'll bet you
can't.

My opinion is that valid blind testing is very difficult

So is quantum mechanics. But just because you can't do it at home
doesn't mean it's wrong.

and no one's
done it yet or even has a good idea how to do it.

That's quite an opinion! Have you tried it out with any professors of
psychology who specialize in perception?

Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.

Only if it's trying to be a test of how we experience music. But DBTs
aren't testing that. They are testing the question, "What sounds do
and do not reach our brains?" Maybe you have a theory about how sounds
that do not reach our brains can nonetheless affect how we experience
music. If so, I'm not sure I want to hear it.

Yet conventional blind testing (ABX with many quick trials for
example) does not control them. Conventional tests assume that the ear/
brain works like measuring equipment. It assumes (1) we experience the
same thing each time we listen,

No, they don't at all. In fact, they control for this by requiring
mulitple trials and/or subjects.

and (2) we can perceive details in
sound independently of our intentions or state of mind.

They don't assume this either. In fact, those conducting good DBTs go
to some lengths to make sure that their subjects are in a state of
mind that is conducive to detecting minute sonic differences.Unlike
you, however, these experts actually know something about that state
of mind.

bob

[MC]

"Mike" wrote in message
...
Just wondering if anyone's interested in discussing the validity of
blind testing; that is, the epistemological underpinning.

My opinion is that valid blind testing is very difficult and no one's
done it yet or even has a good idea how to do it. Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.
Yet conventional blind testing (ABX with many quick trials for
example) does not control them. Conventional tests assume that the ear/
brain works like measuring equipment. It assumes (1) we experience the
same thing each time we listen, and (2) we can perceive details in
sound independently of our intentions or state of mind. But these
things are not true.

I think difference between first and second hearing (for instance) is a
substantial factor in audio tests.

[George Graves]

On Sun, 15 Jul 2007 17:18:52 -0700, Mike wrote
(in article ):

Just wondering if anyone's interested in discussing the validity of
blind testing; that is, the epistemological underpinning.

My opinion is that valid blind testing is very difficult and no one's
done it yet or even has a good idea how to do it. Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.
Yet conventional blind testing (ABX with many quick trials for
example) does not control them. Conventional tests assume that the ear/
brain works like measuring equipment. It assumes (1) we experience the
same thing each time we listen, and (2) we can perceive details in
sound independently of our intentions or state of mind. But these
things are not true.

-Mike

I dunno about the epistemological underpinnings. But I have been on many
double-blind ABX listening panels and have set up more than a few myself. I
always liked to use audiophile and non-audiophile friends, and get a
non-interested third party (read that: wife/girlfriend) to do the switching
after volume is carefully calibrated using a trusty Radio-Shack (or
equivalent) sound level meter. The results were never promising in my
experience. If the volumes were carefully matched, then little or no results
were obtained, and if not, the louder of the components under test, always
was picked as the best.

In my experience, the human perception of music is too complex to yield to
ABX differences and that differences that I couldn't hear in direct
comparison were easily heard on extended listening sessions with each
component (again after carefully matching loudness using a SPL meter).

[Mike]

On Jul 15, 9:07 pm, "MC" wrote:
"Mike" wrote in message

...

Just wondering if anyone's interested in discussing the validity of
blind testing; that is, the epistemological underpinning.

My opinion is that valid blind testing is very difficult and no one's
done it yet or even has a good idea how to do it. Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.
Yet conventional blind testing (ABX with many quick trials for
example) does not control them. Conventional tests assume that the ear/
brain works like measuring equipment. It assumes (1) we experience the
same thing each time we listen, and (2) we can perceive details in
sound independently of our intentions or state of mind. But these
things are not true.

I think difference between first and second hearing (for instance) is a
substantial factor in audio tests.

I agree. The experience of music changes from hearing to hearing.
Composers exploit this when they choose to repeat material sometimes,
and not others. A conventional blind test requires that subjects have
the same experience each time they listen. That may be dozens of
times! That's a huge effect relative to how people experience music.

Mike

[Mike]

On Jul 15, 9:04 pm, Steven Sullivan wrote:
Mike wrote:
Just wondering if anyone's interested in discussing the validity of
blind testing; that is, the epistemological underpinning.
My opinion is that valid blind testing is very difficult and no one's
done it yet or even has a good idea how to do it. Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.
Yet conventional blind testing (ABX with many quick trials for
example) does not control them. Conventional tests assume that the ear/
brain works like measuring equipment. It assumes (1) we experience the
same thing each time we listen, and (2) we can perceive details in
sound independently of our intentions or state of mind. But these
things are not true.

For something that doesn't work, it's odd how well DBT's worked in the
development and improvement of lossy codecs. Not to mention Toole/Olive's
work on speakers. Not to mention experimental perceptual psychology.

A framework produces results which are valid within that framework.
Producing consistent results is not surprising. The question is
whether conventional DBT's provide data on the experience of music,
and how that experience is affected by equipment. I don't object to
"blindness", but I question the validity of most tests.


Btw, your assumptions are off. in 'conventional ABX', you aren't limited
to listening once. You're supposed to listen until you can confidently
make a call that X is either A or B. And no one says you can't repeat the
test.

I didn't assume anything about how many times you listen.

Mike

[Mike]

On Jul 15, 9:06 pm, bob wrote:
On Jul 15, 8:18 pm, Mike wrote:

Just wondering if anyone's interested in discussing the validity of
blind testing; that is, the epistemological underpinning.

Not with someone who doesn't know what epistemology is.

Would you care to tell me what your definition is, rather than making
snipes?

Blind tests, like all scientific tests, are valid to the extent that
they give us reliable, repeatable results that are not inconsistent
with other data. Now, if you can cite data that conflict with blind
test results, we'd have something to talk about. But I'll bet you
can't.

I'm suggesting that you don't have the data I'm interested in. You
have a lot of data on what happens when you treat ears and brains like
measurement instruments. You have no data on how ears and brains
behave under other conditions.


Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.

Only if it's trying to be a test of how we experience music. But DBTs
aren't testing that. They are testing the question, "What sounds do
and do not reach our brains?"

I think even you would disagree with that definition if you thought
about it. They are testing whether two different sounds can be told
apart, or to say it another way, whether they produce two different
experiences. You can't get away from the experience of music unless
you are trying to arbitrarily separate the experience of sound from
the experience of music. That's one poor assumption.

Maybe you have a theory about how sounds
that do not reach our brains can nonetheless affect how we experience
music. If so, I'm not sure I want to hear it.

Yet conventional blind testing (ABX with many quick trials for
example) does not control them. Conventional tests assume that the ear/
brain works like measuring equipment. It assumes (1) we experience the
same thing each time we listen,

No, they don't at all. In fact, they control for this by requiring
mulitple trials and/or subjects.

Exactly.. they require multiple trials while not controlling what's
going on inside your brain during each trial.


and (2) we can perceive details in
sound independently of our intentions or state of mind.

They don't assume this either. In fact, those conducting good DBTs go
to some lengths to make sure that their subjects are in a state of
mind that is conducive to detecting minute sonic differences.

Right.. they have defined "minute sonic differences" as something
different than perceiving music.That definition itself becomes a
limiting factor in how well you can generalize the results.

Unlike
you, however, these experts actually know something about that state
of mind.

Are you interested in discussion or launching insults?

Mike

[Steven Sullivan]

George Graves wrote:
On Sun, 15 Jul 2007 17:18:52 -0700, Mike wrote
(in article ):

Just wondering if anyone's interested in discussing the validity of
blind testing; that is, the epistemological underpinning.

My opinion is that valid blind testing is very difficult and no one's
done it yet or even has a good idea how to do it. Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.
Yet conventional blind testing (ABX with many quick trials for
example) does not control them. Conventional tests assume that the ear/
brain works like measuring equipment. It assumes (1) we experience the
same thing each time we listen, and (2) we can perceive details in
sound independently of our intentions or state of mind. But these
things are not true.

-Mike

I dunno about the epistemological underpinnings. But I have been on many
double-blind ABX listening panels and have set up more than a few myself. I
always liked to use audiophile and non-audiophile friends, and get a
non-interested third party (read that: wife/girlfriend) to do the switching
after volume is carefully calibrated using a trusty Radio-Shack (or
equivalent) sound level meter. The results were never promising in my
experience. If the volumes were carefully matched, then little or no results
were obtained, and if not, the louder of the components under test, always
was picked as the best.

Why are such resuls 'not promising'?

In my experience, the human perception of music is too complex to yield to
ABX differences

is your experience simply anecdotal or has it got any rigorous backing?

and that differences that I couldn't hear in direct
comparison were easily heard on extended listening sessions with each
component (again after carefully matching loudness using a SPL meter).

And, at *that* point, why couldn't you validate the differences with ABX?

It's always puzzling to me why the 'you need extended listening sessions'
faction doesn't perform the obvious 'slam dunk' experiment. If it takes
extended listening to develop a confident perception of difference, fine:
there's no intrinsic 'prohibition' against that in ABX testing. All that's
*non-negotiable* is that, when it comes time to actually demonstrate the
ability to tell the difference, that the choice be made 'blind'.

___
-S
"As human beings, we understand the world through simile, analogy,
metaphor, narrative and, sometimes, claymation." - B. Mason

[Michael Warner]

On 16 Jul 2007 04:08:05 GMT, George Graves wrote:

The results were never promising in my
experience. If the volumes were carefully matched, then little or no results
were obtained

IOW, the audible differences audiophiles claim to hear don't actually
exist. Isn't that great? You can find something more productive
to waste your money on.

ABX differences and that differences that I couldn't hear in direct
comparison were easily heard on extended listening sessions

Yeah. Non-blind listening sessions, I bet :-)

[Walt]

George Graves wrote:

... If the volumes were carefully matched, then little or no results
were obtained, and if not, the louder of the components under test, always
was picked as the best.

Hold on there. The point of a double-blind test is to determine whether
the subject can hear a *difference* between the samples. Not to
determine which one is "best" (whatever that means).

If the volume is not matched, then of course the subject can hear a
difference - "sample A is louder".

//Walt

[Arny Krueger]

"Mike" wrote in message


Just wondering if anyone's interested in discussing the
validity of blind testing; that is, the epistemological
underpinning.

Epistemology is the study of why we believe what we believe, not what we
believe.

My opinion is that valid blind testing is very difficult
and no one's done it yet or even has a good idea how to
do it.

This seems to be more like a discussion of what you believe, than a
discussion of why you believe it.

Are you sure that you know what an epistemological discussion would be like?

Personally, I base as many of my beliefs as I can on my personal
experiences, particularly those experiences which are as unaffected by my
previous state of mind as possible. IOW, I try to have my beliefs about
sonmething rise above my specific prejudices about that thing.

One consequence of this desire is that I prefer to base my beliefs on
bias-controlled tests wherever possible.

Since how we listen, what we listen for, what
we're open to perceiving (and so on) affect our
experience of music, then a valid test would control
these things.

So are you saying you think that we should be open to perceiving that which
is there to be perceived?

Yet conventional blind testing (ABX with
many quick trials for example) does not control them.

Seems like a straw man argument since ABX tests need not involve many quick
trials. Many ABX tests have been done with a few extended trials, for
example.

Conventional tests assume that the ear/ brain works like
measuring equipment.

It's well known that human listening is often very inconsistent, and can be
affected by the listener's beliefs. I know of no test equipment that works
that way. Test equipment tends to be very consistent, and being inanimate,
it has no beliefs at all to bias it.

[bob]

On Jul 16, 6:54 pm, Mike wrote:
On Jul 15, 9:06 pm, bob wrote:

Blind tests, like all scientific tests, are valid to the extent that
they give us reliable, repeatable results that are not inconsistent
with other data. Now, if you can cite data that conflict with blind
test results, we'd have something to talk about. But I'll bet you
can't.

I'm suggesting that you don't have the data I'm interested in.

So you just ignore it? That's hardly the appropriate posture for
someone who claims to want a discussion. And at least I HAVE data.

You
have a lot of data on what happens when you treat ears and brains like
measurement instruments. You have no data on how ears and brains
behave under other conditions.

You are playing semantic games here. ABX subjects are using their ears
to *hear.* If you think that hearing operates differently under
different conditions, then it is incumbent on you to a) define what
those conditions are, and b) provide evidence that it does, indeed,
operate differently. You haven't--and cannot--do that. So what's to
discuss?

Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.

Only if it's trying to be a test of how we experience music. But DBTs
aren't testing that. They are testing the question, "What sounds do
and do not reach our brains?"

I think even you would disagree with that definition if you thought
about it. They are testing whether two different sounds can be told
apart,

Which is another way of stating what I said.

or to say it another way, whether they produce two different
experiences. You can't get away from the experience of music unless
you are trying to arbitrarily separate the experience of sound from
the experience of music. That's one poor assumption.

I'm not "getting away from the experience of music." You're the one
making the claims--and the wild assumptions--here. You're assuming, or
claiming, that "experiencing music" will somehow change the
sensitivity of our hearing. But you have no evidence for this. Indeed,
available evidence suggests the opposite--that extended music
listening is a LESS sensitive way to test for audible differences.

bob

[George Graves]

On Mon, 16 Jul 2007 15:52:52 -0700, ScottW wrote
(in article ):

"George Graves" wrote in message
...
On Sun, 15 Jul 2007 17:18:52 -0700, Mike wrote
(in article ):

Just wondering if anyone's interested in discussing the validity of
blind testing; that is, the epistemological underpinning.

My opinion is that valid blind testing is very difficult and no one's
done it yet or even has a good idea how to do it. Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.
Yet conventional blind testing (ABX with many quick trials for
example) does not control them. Conventional tests assume that the ear/
brain works like measuring equipment. It assumes (1) we experience the
same thing each time we listen, and (2) we can perceive details in
sound independently of our intentions or state of mind. But these
things are not true.

-Mike

I dunno about the epistemological underpinnings. But I have been on many
double-blind ABX listening panels and have set up more than a few myself. I
always liked to use audiophile and non-audiophile friends, and get a
non-interested third party (read that: wife/girlfriend) to do the switching
after volume is carefully calibrated using a trusty Radio-Shack (or
equivalent) sound level meter. The results were never promising in my
experience. If the volumes were carefully matched, then little or no results
were obtained, and if not, the louder of the components under test, always
was picked as the best.

In my experience, the human perception of music is too complex to yield to
ABX differences and that differences that I couldn't hear in direct
comparison were easily heard on extended listening sessions

Is there a reason ABX couldn't include extended listening
sessions?

ScottW

No, it's just that I would expect that one would get a better idea of the
true sound of the DUT without the associated switching apparatus which
accompanies ABX testing.

[George Graves]

On Mon, 16 Jul 2007 15:55:20 -0700, Steven Sullivan wrote
(in article ):

George Graves wrote:
On Sun, 15 Jul 2007 17:18:52 -0700, Mike wrote
(in article ):

Just wondering if anyone's interested in discussing the validity of
blind testing; that is, the epistemological underpinning.

My opinion is that valid blind testing is very difficult and no one's
done it yet or even has a good idea how to do it. Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.
Yet conventional blind testing (ABX with many quick trials for
example) does not control them. Conventional tests assume that the ear/
brain works like measuring equipment. It assumes (1) we experience the
same thing each time we listen, and (2) we can perceive details in
sound independently of our intentions or state of mind. But these
things are not true.

-Mike

I dunno about the epistemological underpinnings. But I have been on many
double-blind ABX listening panels and have set up more than a few myself. I
always liked to use audiophile and non-audiophile friends, and get a
non-interested third party (read that: wife/girlfriend) to do the switching
after volume is carefully calibrated using a trusty Radio-Shack (or
equivalent) sound level meter. The results were never promising in my
experience. If the volumes were carefully matched, then little or no
results
were obtained, and if not, the louder of the components under test, always
was picked as the best.

Why are such resuls 'not promising'?

In my experience, the human perception of music is too complex to yield to
ABX differences

is your experience simply anecdotal or has it got any rigorous backing?

and that differences that I couldn't hear in direct
comparison were easily heard on extended listening sessions with each
component (again after carefully matching loudness using a SPL meter).

And, at *that* point, why couldn't you validate the differences with ABX?

It's always puzzling to me why the 'you need extended listening sessions'
faction doesn't perform the obvious 'slam dunk' experiment. If it takes
extended listening to develop a confident perception of difference, fine:
there's no intrinsic 'prohibition' against that in ABX testing. All that's
*non-negotiable* is that, when it comes time to actually demonstrate the
ability to tell the difference, that the choice be made 'blind'.

I'm sorry. I simply cannot answer your questions. I'm merely going by my own
experience as a listener. And this argument about subjective vs objective
testing methodologies have been going on much longer than we've been
discussing it, and nobody has ever, to my knowledge been able resolve the
argument. I used believe explicitly in objective ABX testing until I became
absolutely convinced that it really wasn't very good at characterizing the
differences that one hears between components.

[George Graves]

On Mon, 16 Jul 2007 15:56:01 -0700, Michael Warner wrote
(in article ):

On 16 Jul 2007 04:08:05 GMT, George Graves wrote:

The results were never promising in my
experience. If the volumes were carefully matched, then little or no
results
were obtained

IOW, the audible differences audiophiles claim to hear don't actually
exist. Isn't that great? You can find something more productive
to waste your money on.

ABX differences and that differences that I couldn't hear in direct
comparison were easily heard on extended listening sessions

Yeah. Non-blind listening sessions, I bet :-)

And it could be otherwise, how? I'm the one replacing my reference component
with the DUT. Of course its non-blind.

[Harry Lavo]

"Walt" wrote in message
...
George Graves wrote:

... If the volumes were carefully matched, then little or no results were
obtained, and if not, the louder of the components under test, always was
picked as the best.

Hold on there. The point of a double-blind test is to determine whether
the subject can hear a *difference* between the samples. Not to determine
which one is "best" (whatever that means).

If the volume is not matched, then of course the subject can hear a
difference - "sample A is louder".

//Walt

No, that is the point of an ABX test. It is perfectly scientific to have a
blind A-B preference test. Blindness is a test condition applicable to many
different types of tests.

[bob]

On Jul 16, 12:08 am, George Graves wrote:
In my experience, the human perception of music is too complex to yield to
ABX differences and that differences that I couldn't hear in direct
comparison were easily heard on extended listening sessions with each
component (again after carefully matching loudness using a SPL meter).

In other words, you changed not one but two conditions of your
comparison:

1) how long you listened, and

2) how much visual information you had

Then you assumed that the differing result came because of #1. The
fallacy of this reasoning should be obvious.

bob

[Steven Sullivan]

Mike wrote:
On Jul 15, 9:07 pm, "MC" wrote:
"Mike" wrote in message

...

Just wondering if anyone's interested in discussing the validity of
blind testing; that is, the epistemological underpinning.

My opinion is that valid blind testing is very difficult and no one's
done it yet or even has a good idea how to do it. Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.
Yet conventional blind testing (ABX with many quick trials for
example) does not control them. Conventional tests assume that the ear/
brain works like measuring equipment. It assumes (1) we experience the
same thing each time we listen, and (2) we can perceive details in
sound independently of our intentions or state of mind. But these
things are not true.

I think difference between first and second hearing (for instance) is a
substantial factor in audio tests.

I agree. The experience of music changes from hearing to hearing.
Composers exploit this when they choose to repeat material sometimes,
and not others. A conventional blind test requires that subjects have
the same experience each time they listen. That may be dozens of
times! That's a huge effect relative to how people experience music.

These analogies keep getting sillier and sillier.

A 'conventional blind test' requires nothing of the sort. It requires
that the subject be 'blinded' to the source of the sound -- and thus use
only their ears, to judge what they're hearing. That's all, essentially.
Every recommedation beyond that is geared to *increasing* the subject's
chances of discerning an audible difference.

Instead of hypothesizing bogus 'problems' with blind testing, how about
acknowledging the intrinsic, inescapable, well-established issues
accompanying ANY 'sighted' evaluation of audio?

___
-S
"As human beings, we understand the world through simile, analogy,
metaphor, narrative and, sometimes, claymation." - B. Mason

[Steven Sullivan]

Mike wrote:
and do not reach our brains?"

I think even you would disagree with that definition if you thought
about it. They are testing whether two different sounds can be told
apart, or to say it another way, whether they produce two different
experiences. You can't get away from the experience of music unless
you are trying to arbitrarily separate the experience of sound from
the experience of music. That's one poor assumption.

I can listen to the eact same recorded performance of a work on the exact
same rig -- even minutes apart -- and have different 'experiences' of it.
For example, I might be more bored by it the second time. But of course
the *sound* hasn't changed at all. Yet by your logic, I've listened to
two different sounds.

Imagine if that logic were carried over into all scientific work.

Maybe you have a theory about how sounds
that do not reach our brains can nonetheless affect how we experience
music. If so, I'm not sure I want to hear it.

Yet conventional blind testing (ABX with many quick trials for
example) does not control them. Conventional tests assume that the ear/
brain works like measuring equipment. It assumes (1) we experience the
same thing each time we listen,

No, they don't at all. In fact, they control for this by requiring
mulitple trials and/or subjects.

Exactly.. they require multiple trials while not controlling what's
going on inside your brain during each trial.

That's the POINT. Your *brain* will quite readily 'hear' difference where
*none exists*. The old 'phantom switch' setup is a classic example, where
literally nothing is changed, but the listener is led to believe that
something has been changed. Usually they report hearing 'differnece' --
sometimes a comically large one.

So, let the brain 'do' what it likes. Running the comparison 'blind'
takes care of its tendency to overestimate the occurence of 'difference'.
Meanwhile, with training, some tiny measurable differences -- e.g., on the
order of 0.2 dB in some frequency ranges -- can be confidently shown to be
discernable via DBT. Others...not so much. Which makes sense; no one
should expect that all measurable differences are perceptible (whereas all
perceptible differences so far, have turned out to have a measurement
affected by them, indirectly if not directly).

and (2) we can perceive details in
sound independently of our intentions or state of mind.

They don't assume this either. In fact, those conducting good DBTs go
to some lengths to make sure that their subjects are in a state of
mind that is conducive to detecting minute sonic differences.

Right.. they have defined "minute sonic differences" as something
different than perceiving music.That definition itself becomes a
limiting factor in how well you can generalize the results.

"perceived" difference in music, as for any sound, can occur even when the
music itself is exactly the same. How do you propose to get around this
fundamental problem?

___
-S
"As human beings, we understand the world through simile, analogy,
metaphor, narrative and, sometimes, claymation." - B. Mason

[Steven Sullivan]

Walt wrote:
George Graves wrote:

... If the volumes were carefully matched, then little or no results
were obtained, and if not, the louder of the components under test, always
was picked as the best.

Hold on there. The point of a double-blind test is to determine whether
the subject can hear a *difference* between the samples. Not to
determine which one is "best" (whatever that means).

DBT can be used to determine if preference is due to sound, or
to something else. Of course, it will already have been determined
that the two things sound *different*. Such is the case for
using DBTs to study loudspeaker preference.

___
-S
"As human beings, we understand the world through simile, analogy,
metaphor, narrative and, sometimes, claymation." - B. Mason

[Steven Sullivan]

George Graves wrote:
On Mon, 16 Jul 2007 15:52:52 -0700, ScottW wrote
(in article ):

"George Graves" wrote in message
...
On Sun, 15 Jul 2007 17:18:52 -0700, Mike wrote
(in article ):

Just wondering if anyone's interested in discussing the validity of
blind testing; that is, the epistemological underpinning.

My opinion is that valid blind testing is very difficult and no one's
done it yet or even has a good idea how to do it. Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.
Yet conventional blind testing (ABX with many quick trials for
example) does not control them. Conventional tests assume that the ear/
brain works like measuring equipment. It assumes (1) we experience the
same thing each time we listen, and (2) we can perceive details in
sound independently of our intentions or state of mind. But these
things are not true.

-Mike

I dunno about the epistemological underpinnings. But I have been on many
double-blind ABX listening panels and have set up more than a few myself. I
always liked to use audiophile and non-audiophile friends, and get a
non-interested third party (read that: wife/girlfriend) to do the switching
after volume is carefully calibrated using a trusty Radio-Shack (or
equivalent) sound level meter. The results were never promising in my
experience. If the volumes were carefully matched, then little or no results
were obtained, and if not, the louder of the components under test, always
was picked as the best.

In my experience, the human perception of music is too complex to yield to
ABX differences and that differences that I couldn't hear in direct
comparison were easily heard on extended listening sessions

Is there a reason ABX couldn't include extended listening
sessions?

ScottW

No, it's just that I would expect that one would get a better idea of the
true sound of the DUT without the associated switching apparatus which
accompanies ABX testing.

That's not necessarily a well-warranted expectation -- the 'switch' can range from manually changing the source, to
using a sophisticated device like the ABX Comparator -- but by all means, if you think a switch interfered with 'getting
a better idea of the true sound', then listen *without the switch* until you are sure you can identify the true sound --
*then* do the ABX!

___
-S
"As human beings, we understand the world through simile, analogy,
metaphor, narrative and, sometimes, claymation." - B. Mason

[Steven Sullivan]

George Graves wrote:
On Mon, 16 Jul 2007 15:55:20 -0700, Steven Sullivan wrote
faction doesn't perform the obvious 'slam dunk' experiment. If it takes
extended listening to develop a confident perception of difference, fine:
there's no intrinsic 'prohibition' against that in ABX testing. All that's
*non-negotiable* is that, when it comes time to actually demonstrate the
ability to tell the difference, that the choice be made 'blind'.

I'm sorry. I simply cannot answer your questions. I'm merely going by my own
experience as a listener. And this argument about subjective vs objective
testing methodologies have been going on much longer than we've been
discussing it, and nobody has ever, to my knowledge been able resolve the
argument. I used believe explicitly in objective ABX testing until I became
absolutely convinced that it really wasn't very good at characterizing the
differences that one hears between components.

It sounds like you ahve 'resolved the argument' in your own mind -- based on
*what*? A 'slam-dunk' experiment as I described, or you have one of your
own? Or was it just, again, sighted comparison -- a 'protocol' whose
flaws are not seriously questioned by researchers into perception...or
even orchestra audition panels.

___
-S
"As human beings, we understand the world through simile, analogy,
metaphor, narrative and, sometimes, claymation." - B. Mason

[George Graves]

On Mon, 16 Jul 2007 20:30:19 -0700, Harry Lavo wrote
(in article ):

"Walt" wrote in message
...
George Graves wrote:

... If the volumes were carefully matched, then little or no results were
obtained, and if not, the louder of the components under test, always was
picked as the best.

Hold on there. The point of a double-blind test is to determine whether
the subject can hear a *difference* between the samples. Not to determine
which one is "best" (whatever that means).

If the volume is not matched, then of course the subject can hear a
difference - "sample A is louder".

//Walt

No, that is the point of an ABX test. It is perfectly scientific to have a
blind A-B preference test. Blindness is a test condition applicable to many
different types of tests.


True. It just doesn't seem applicable to audio and the way people perceive
sound.

[George Graves]

On Mon, 16 Jul 2007 20:31:39 -0700, bob wrote
(in article ):

On Jul 16, 12:08 am, George Graves wrote:
In my experience, the human perception of music is too complex to yield to
ABX differences and that differences that I couldn't hear in direct
comparison were easily heard on extended listening sessions with each
component (again after carefully matching loudness using a SPL meter).

In other words, you changed not one but two conditions of your
comparison:

1) how long you listened, and

2) how much visual information you had

Then you assumed that the differing result came because of #1. The
fallacy of this reasoning should be obvious.

bob

Both of your assumptions are just that. But I'm not here to argue this point.
You want ABX comparisons? Fine. You pick your audio system based on the
criteria you find important, and I pick my components based upon how the
equipment sounds to me. What's interesting is that when we "subjective
testers" get together, we all seem to find the same characteristics for the
same equipment, even though we all came by these "sonic signatures"
independently of one another on our own systems and they usually don't show
up under ABX testing conditions. Mass hysteria? You tell me.

[Arny Krueger]

"Mike" wrote in message


I agree. The experience of music changes from hearing to
hearing.

I think you mean that we experience music differently every time we hear it.

Composers exploit this when they choose to
repeat material sometimes, and not others.

OK.

A conventional
blind test requires that subjects have the same
experience each time they listen.

Wrong. In fact properly designed listening tests account for the fact that
we have an at least slightly different experience every time we hear the
same segment of music.

It is conventional audiophile sighted listening tests that are based on the
idea that subjects have the same
experience each time they listen.

[Arny Krueger]

"George Graves" wrote in message


No, it's just that I would expect that one would get a
better idea of the true sound of the DUT without the
associated switching apparatus which accompanies ABX
testing.

In fact it is relatively easy to do bias-controlled including ABX listening
tests with no extra switching apparatus at all.

[Arny Krueger]

"George Graves" wrote in message


I'm sorry. I simply cannot answer your questions. I'm
merely going by my own experience as a listener.

Does that experience include actually doing any bias-controlled listening
tests?

And this
argument about subjective vs objective testing
methodologies have been going on much longer than we've
been discussing it, and nobody has ever, to my knowledge
been able resolve the argument.

Actually, the argument has been resolved for a long time. Anybody who wants
to do a serious listening test, that is one where the results have a great
deal of credibility, does their tests with appropriate bias controls in
place.

I used believe explicitly
in objective ABX testing until I became absolutely
convinced that it really wasn't very good at
characterizing the differences that one hears between
components.

How did you become convinced that way?

Based on your posts, you became convinced of a lot things that are not true
along the way.

[Arny Krueger]

"George Graves" wrote in message

On Mon, 16 Jul 2007 15:56:01 -0700, Michael Warner wrote
(in article ):

On 16 Jul 2007 04:08:05 GMT, George Graves wrote:

The results were never promising in my
experience. If the volumes were carefully matched, then
little or no results
were obtained

IOW, the audible differences audiophiles claim to hear
don't actually exist. Isn't that great? You can find
something more productive
to waste your money on.

ABX differences and that differences that I couldn't
hear in direct comparison were easily heard on extended
listening sessions

Yeah. Non-blind listening sessions, I bet :-)

I might question the level matching.

And it could be otherwise, how? I'm the one replacing my
reference component with the DUT. Of course its
non-blind.

Long term listening tests have been done where the replacement process was
hidden from the listener, and randomized.

[[email protected]]

Arny Krueger wrote:

Epistemology is the study of *why* we believe what we believe, not what we
believe.

I think it would be more correct to say that epistemology is the science of
the limits of knowledge, and, in a way, how we can ascertain knowledge
within these limits (method). I see it more as the "what" than the "why."
Speaking loosely, "why" we believe encompasses the psychology behind
belief--something a bit different from epistemology proper. This also turns
on a distinction between belief and knowledge. The latter implies the
former, but it is not necessarily the other way around.

mp

[MC]

"Arny Krueger" wrote in message
...

A conventional
blind test requires that subjects have the same
experience each time they listen.

Wrong. In fact properly designed listening tests account for the fact that
we have an at least slightly different experience every time we hear the
same segment of music.

Right. A well-designed double-blind test will try the experiment several
times, varying the order.

[Mike]

On Jul 16, 4:00 pm, "Arny Krueger" wrote:
"Mike" wrote in message


Conventional tests assume that the ear/ brain works like
measuring equipment.

It's well known that human listening is often very inconsistent, and can be
affected by the listener's beliefs. I know of no test equipment that works
that way. Test equipment tends to be very consistent, and being inanimate,
it has no beliefs at all to bias it.

Exactly, that's why it is such a poor assumption that people
experience the same thing each time they listen to a section (small or
large) of music. I've never heard of an ABX test that didn't assume
that.

Mike

[Mike]

On Jul 16, 8:26 pm, bob wrote:
On Jul 16, 6:54 pm, Mike wrote:

On Jul 15, 9:06 pm, bob wrote:

You
have a lot of data on what happens when you treat ears and brains like
measurement instruments. You have no data on how ears and brains
behave under other conditions.

You are playing semantic games here. ABX subjects are using their ears
to *hear.* If you think that hearing operates differently under
different conditions, then it is incumbent on you to a) define what
those conditions are, and b) provide evidence that it does, indeed,
operate differently. You haven't--and cannot--do that. So what's to
discuss?

I would like to discuss what assumptions one has to make about the ear/
brain to investigate it. You write as though this were some kind of
personal battle of opposing viewpoints. You also seem to need things
to be precisely defined in order to believe they exist. It's obvious
hearing operates differently under different conditions. For example,
you have many choices about _what_ to listen for, and you can easily
choose one thing but miss something else, so your intention is one
condition.

But every blind test I'm aware of makes the assumption that we have
the same experience each time we listen to something. That's a very
poor assumption, and that makes the results suspect. If it's incumbent
on anyone to provide proof, it's incumbent on those who do the
experiments to show that they have a firm basis.


Since how we listen,
what we listen for, what we're open to perceiving (and so on) affect
our experience of music, then a valid test would control these things.

Only if it's trying to be a test of how we experience music. But DBTs
aren't testing that. They are testing the question, "What sounds do
and do not reach our brains?"

I think even you would disagree with that definition if you thought
about it. They are testing whether two different sounds can be told
apart,

Which is another way of stating what I said.

Your statement is very loose, though. It implies we are testing
whether a signal is "large enough" to reach our brains, which we are
not testing. Both A and B reach our brains.


or to say it another way, whether they produce two different
experiences. You can't get away from the experience of music unless
you are trying to arbitrarily separate the experience of sound from
the experience of music. That's one poor assumption.

I'm not "getting away from the experience of music." You're the one
making the claims--and the wild assumptions--here. You're assuming, or
claiming, that "experiencing music" will somehow change the
sensitivity of our hearing. But you have no evidence for this.

You're assuming it does not change the sensitivity, and provide no
evidence for that.

Indeed,
available evidence suggests the opposite--that extended music
listening is a LESS sensitive way to test for audible differences.

"Extended" vs "non-extended" is not the point. Even "enjoying music"
is not the point. The point is that our experience as a whole changes
each time we listen.

Mike

[George Graves]

On Tue, 17 Jul 2007 15:55:51 -0700, Steven Sullivan wrote
(in article ):

George Graves wrote:
On Mon, 16 Jul 2007 15:55:20 -0700, Steven Sullivan wrote
faction doesn't perform the obvious 'slam dunk' experiment. If it takes
extended listening to develop a confident perception of difference, fine:
there's no intrinsic 'prohibition' against that in ABX testing. All that's
*non-negotiable* is that, when it comes time to actually demonstrate the
ability to tell the difference, that the choice be made 'blind'.

I'm sorry. I simply cannot answer your questions. I'm merely going by my
own
experience as a listener. And this argument about subjective vs objective
testing methodologies have been going on much longer than we've been
discussing it, and nobody has ever, to my knowledge been able resolve the
argument. I used believe explicitly in objective ABX testing until I became
absolutely convinced that it really wasn't very good at characterizing the
differences that one hears between components.

It sounds like you ahve 'resolved the argument' in your own mind -- based on
*what*? A 'slam-dunk' experiment as I described, or you have one of your
own? Or was it just, again, sighted comparison -- a 'protocol' whose
flaws are not seriously questioned by researchers into perception...or
even orchestra audition panels.

What difference does it make? I'm not trying to convince you (or anyone else)
of anything. I merely stated what my 40 years of being an audio enthusiast
and 30 years of writing about it have told me. You don't accept it? That's
OK. It's up to you to come to your own conclusions, and in the final
analysis, what anyone else finds to be true on such a subjective issue as
sound is totally irrelevant.

[George Graves]

On Tue, 17 Jul 2007 16:02:07 -0700, Arny Krueger wrote
(in article ):

"George Graves" wrote in message

On Mon, 16 Jul 2007 15:56:01 -0700, Michael Warner wrote
(in article ):

On 16 Jul 2007 04:08:05 GMT, George Graves wrote:

The results were never promising in my
experience. If the volumes were carefully matched, then
little or no results
were obtained

IOW, the audible differences audiophiles claim to hear
don't actually exist. Isn't that great? You can find
something more productive
to waste your money on.

ABX differences and that differences that I couldn't
hear in direct comparison were easily heard on extended
listening sessions

Yeah. Non-blind listening sessions, I bet :-)

I might question the level matching.

And it could be otherwise, how? I'm the one replacing my
reference component with the DUT. Of course its
non-blind.

Long term listening tests have been done where the replacement process was
hidden from the listener, and randomized.


Yeah, I've done them too. It works. I can listen to a component where I don't
know what I'm listening too and make comments that characterize the sound I
hear. But what of it? Am I commenting on one component or on the entire
playback chain. If I'm not intimately familiar with the sound of the entire
system, then it's obvious that my findings are about the latter. Not too
useful. But if someone comes into my home, and replaces a component blindly
so that I don't know what it is, and I use material with which I'm familiar
with which to listen, then any changes from what I normally hear are
obviously going to be attributable to whatever component was exchanged (or
the way in which the "foreign" component interacts with the rest of my
system). Wouldn't this make sense?

[Arny Krueger]

"George Graves" wrote in message


What's interesting is that when
we "subjective testers" get together, we all seem to find
the same characteristics for the same equipment, even
though we all came by these "sonic signatures"
independently of one another on our own systems.

True, that subjective reviewers tend to draw their descriptive words and
phrases from a relatively small pool, and there's a tendency for subjective
reviews to all sound the same.

[Arny Krueger]

"Walt" wrote in message

George Graves wrote:

... If the volumes were carefully matched, then little
or no results were obtained, and if not, the louder of the components
under test, always was picked as the best.

Hold on there. The point of a double-blind test is to
determine whether the subject can hear a *difference*
between the samples.

Not necessarily.

The point of a double-blind test is to control the parameters that are
apparent to the listener. The basic test can be a wide variety of tests,
including preference testing.

One of the most common kind of double blind listening tests is called the
ABC/hr test which is a test of perceived level of degradation. While not
exactly a preference test, and while it can yield information about the
listener's ability to hear differences, it is akin to a preference test,
presuming that listeners prefer the original sound.

Not to determine which one is "best"
(whatever that means).

If there is an agreement about audiophiles and professionals about what
sounds best, the most universal stated preference is for uncolored sound
that is as much as like the original sound.

If the volume is not matched, then of course the subject
can hear a difference - "sample A is louder".

No doubt. All listening tests should be in some sense level-matched.
However, if the equipment being compared has significant colorations, then
the true meaning of level-matching becomes less clear.

I'll bet money Walt that you have never before heard of ABC/hr listening
tests, just like you obviously never ever before heard about long-term ABX
tests. This means that you might feel the need to study up some more before
making any more global pronouncements.

[Walt]

George Graves wrote:
On Mon, 16 Jul 2007 20:31:39 -0700, bob wrote
On Jul 16, 12:08 am, George Graves wrote:

In my experience, the human perception of music is too complex to yield to
ABX differences and that differences that I couldn't hear in direct
comparison were easily heard on extended listening sessions with each
component (again after carefully matching loudness using a SPL meter).

In other words, you changed not one but two conditions of your
comparison:

1) how long you listened, and

2) how much visual information you had

Then you assumed that the differing result came because of #1. The
fallacy of this reasoning should be obvious.


Both of your assumptions are just that. But I'm not here to argue this point.
You want ABX comparisons? Fine. You pick your audio system based on the
criteria you find important, and I pick my components based upon how the
equipment sounds to me. What's interesting is that when we "subjective
testers" get together, we all seem to find the same characteristics for the
same equipment, even though we all came by these "sonic signatures"
independently of one another on our own systems and they usually don't show
up under ABX testing conditions. Mass hysteria? You tell me.

This seems to be an example of a scientifically well-established and
very common effect:
See http://en.wikipedia.org/wiki/Asch_co...ty_experiments

If your opinion was entirely based on your own audio perception, it
would be apparent in double blind listening tests. Since it isn't,
there must be some non-auditory phenomenon at work. Dr. Occam and I vote
for the Asch effect.

//Walt

[George Graves]

On Wed, 18 Jul 2007 15:55:58 -0700, Arny Krueger wrote
(in article ):

"George Graves" wrote in message


What's interesting is that when
we "subjective testers" get together, we all seem to find
the same characteristics for the same equipment, even
though we all came by these "sonic signatures"
independently of one another on our own systems.

True, that subjective reviewers tend to draw their descriptive words and
phrases from a relatively small pool, and there's a tendency for subjective
reviews to all sound the same.


Nice rationalization, but you're overlooking something. If I say that a
certain pre-amp has a sandpaper-like dryness in the upper octaves as opposed
to being syruppy, or dark, or grainy, and other subjective reviewers come to
the same conclusion independently, then, it would almost have to follow that
this particular component, does, indeed, exhibit that characteristic, would
it not?

[George Graves]

On Wed, 18 Jul 2007 16:01:22 -0700, Walt wrote
(in article ):

George Graves wrote:
On Mon, 16 Jul 2007 20:31:39 -0700, bob wrote
On Jul 16, 12:08 am, George Graves wrote:

In my experience, the human perception of music is too complex to yield to
ABX differences and that differences that I couldn't hear in direct
comparison were easily heard on extended listening sessions with each
component (again after carefully matching loudness using a SPL meter).

In other words, you changed not one but two conditions of your
comparison:

1) how long you listened, and

2) how much visual information you had

Then you assumed that the differing result came because of #1. The
fallacy of this reasoning should be obvious.


Both of your assumptions are just that. But I'm not here to argue this
point.
You want ABX comparisons? Fine. You pick your audio system based on the
criteria you find important, and I pick my components based upon how the
equipment sounds to me. What's interesting is that when we "subjective
testers" get together, we all seem to find the same characteristics for the
same equipment, even though we all came by these "sonic signatures"
independently of one another on our own systems and they usually don't show
up under ABX testing conditions. Mass hysteria? You tell me.

This seems to be an example of a scientifically well-established and
very common effect:
See http://en.wikipedia.org/wiki/Asch_co...ty_experiments

If your opinion was entirely based on your own audio perception, it
would be apparent in double blind listening tests. Since it isn't,
there must be some non-auditory phenomenon at work. Dr. Occam and I vote
for the Asch effect.

How can the Asch effect be an issue, when another reviewer who I do not know
in another publication comes to exactly the same conclusions that I have come
to, when neither of us knew that the other was auditioning the same piece of
equipment?

I cannot speak for others, of course, but, my audio perception is based upon
my memory or real, live music. I go to concerts constantly. I frequent local
jazz clubs, attend concerts of symphonic as well as chamber music several
times a week. While I'm sure that my memory of live music is imperfect, it
makes me a better judge, I think, than someone who merely compares one audio
component to another, relying on nothing for a touchstone to reality or a
reference. Musical perception is extremely complex and many things can change
the sound of music from one listening instance to the next, but I do know
that there are certain instrumental "signatures" that manage to survive the
differences imposed by venue acoustics, relative seating location, etc. and I
try to cue-in on those.

I also make my own recordings of many of these. I use a MiniDisc Hi-MD
recorder in the 16-bit linear record mode and a Sony LT-929 'MS' stereo
microphone for many of them (especially jazz club performance - and yes, I
always ask permission). When I use these recordings as well as symphonic
recordings that I have made using professional equipment as my source
materials when evaluating equipment, I can easily tell when a piece of
equipment has introduced some coloration that is not consonant with the sound
of live music.

None of this is perfect, to say the least, but it does make my perceptions of
a piece of equipment's sonic signature similar to that of other reviewers
auditioning the same make and model piece of equipment. This has happened to
me so many times that I know its not coincidence. I think that John Atkinson
of Stereophile posts here occasionally. I believe that if you ask him, he''ll
tell you that the same thing has happened to him, probably countless times.
It's not a rare occurrence. This validates long-term listening tests vs ABX
tests as far as I'm concerned.