Darryl Miyaguchi wrote:

On 3 Jul 2003 02:52:02 GMT, (Nousaine) wrote:

Perhaps we're missing each others points here. With the common ABC/Hr
protocol
the listener is asked to identify one alternative as being the same as the
reference and to grade the alternative on a downward rating scale.

If I'm taking your point correctly those listeners that reliably rate an
alternative as being worse than itself have responses thrown out. If so,
this
could be a conscious form of subject bias.

But it could also be a form of experimental bias where some form of other
identification cue is involved. In either case SOME form of bias is present;
just as 'reverse' significant results in common A/B Same/Different or even
ABX
tests would indicate.

Backward significance is always an indictator that some form of bias is
present.

Just so that we don't misunderstand exactly what is happening during
one of these comparative tests:

Note the implementation of ABC/Hr being used:

http://ff123.net/abchr/abchr.html

The reference is not being presented as another codec. In the process
of rating each codec, the listener must choose from the hidden
reference.

Yes, the hidden reference is the original unprocessed source.

This is different from MUSHRA, where there is a hidden
reference, but which is typically presented as another codec. I think
we're both on the same page when talking about ABC/Hr, but I just want
to make sure.

When I speak of a bias, I am mostly concerned that one codec is not
falsely preferred over another.

In the ABC/Hr implementation we're discussing, suppose a listener
downrates the hidden reference for one or more codecs. Such a
listener is not an *outlier* -- he is clearly *incorrect*. Then, the
procedure used would be to discard that particular listener's entire
set of results for all codecs.

I don't believe there is a preferential bias of the type I am
concerned about. Instead, I believe that what is happening is that
statistical noise is being reduced when unreliable listeners are
removed, but possibly sensitivity is being reduced as well for the
reasons I outlined in my previous post. Reduced sensitivity is ok for
the purposes of this experiment; preferential bias is not.

Actually in the ABC/Hr protocol any listener reliably rating the hidden
reference worse than itself has demonstrated SOME form of test or subject bias.
Ther eis no other explanation; either the protocol has some form of non-sonic
confounding identifier OR the subject can truly hear the difference and is
purposefully responding in a backward manner.

I'm all for rejecting biased data but any kind of significant "reverse" results
shoyld be followed by an examination of the experiment to find whether it's
specific to a given subject(s). If the latter cannot be shown then the entire
experiment can be ruled invalid.


Now, one could argue that this procedure is selecting a special group
out of the test population. If one of the purposes of the experiment
is to represent a certain listening population, then throwing out
unreliable listeners is changing things. However, this particular
test makes no pretensions of representing the average listener. The
people who participate in this type of experiment are already
self-selected, and likely to be more sensitive than the average Joe.

Most likely true.


Is there some mechanism whereby you think that some sort of bias
(falsely preferring one or more codecs over others) may be operating
if unreliable results of the type described are thrown out? I can't
think of any.

Darryl Miyaguchi

As I said earlier if you cannot determine that results were limited to a given
subject(s) then the whole experiment must be considered suspect. At the very
least it should be repeated.