Just so that we don't misunderstand exactly what is happening during
one of these comparative tests:

Note the implementation of ABC/Hr being used:

http://ff123.net/abchr/abchr.html

The reference is not being presented as another codec. In the process
of rating each codec, the listener must choose from the hidden
reference. This is different from MUSHRA, where there is a hidden
reference, but which is typically presented as another codec. I think
we're both on the same page when talking about ABC/Hr, but I just want
to make sure.

When I speak of a bias, I am mostly concerned that one codec is not
falsely preferred over another.

In the ABC/Hr implementation we're discussing, suppose a listener
downrates the hidden reference for one or more codecs. Such a
listener is not an *outlier* -- he is clearly *incorrect*. Then, the
procedure used would be to discard that particular listener's entire
set of results for all codecs.

After checking the test description, I see that the test has a built
in assumption that the reference is best, and asks the subject to
rate the codec in terms of the amount of "annoying degredation".
If a subject is consistently rating a hidden reference as degraded
under those conditions, I would think that what's happening is this:

The subject hears that particular codec as euphonic. That doesn't fit
the test assumption, and there isn't any way to rate the codec as better
than the reference. The easy way out of that dilemma, rather than
thinking about it enough to recognize what has happened,
is to assume the reference is the better one as the tester
says it should be, and if the subject thinks otherwise they must
have misidentified it. So they pick the better one as the reference
and report degradation on the other one.

If you discard the data from the people who did this, rather than
recognizing what it means, wouldn't that create a severe bias in this
testing procedure against any codec that sounds euphonic to some
people?

Bob