[bob]

On Dec 14, 1:01 pm, "Arny Krueger" wrote:

I pasted their matrix into Excel and tried to do some quick sums. I came up
with Test Condition 1 = 23/60 and Test Condition 2 = 31/60.

To be precise, Condition 2 is 31/54, since one subject's results were
not available. (It happens.)

IOW, one test
produced an outcome that was worse than random guessing, and the other was
random guessing.

Actually, both could be random guessing, since neither hits even the
90% confidence level.

Results that are worse than random guessing may cause some head scratching,
but they are not all that unusual in experiments like this where
communication between the listeners can affect the outcome.

Not the case here, where each subject was tested individually. Also,
unlike an ABX test, there is no "wrong" answer here. Either conversion
could be judged better. (Unless, of course, you're Philips and you
sell hi-rez converters.)

The most probable explanation for worse-than-random results is that that
some of the listeners were basing their results on their perceptions of what
other listeners were perceiving, and the total number of independent
responses was far less than what you get from a naive count of the actual
responses.

IOW, the actual situation was not 23 independent responses of 60 trials, but
maybe more like 4 independent responses of 10 independent trials, so the
true numbers were so small that statistics doesn't really apply.

Again, this is an inaccurate description of the actual test. The
responses were independent.

I think these results would pretty well explain themselves to just about
anybody, were they reproduced any place within the actual paper.

Short answer - the outcome was random guessing, and both the test itself and
the statistical analysis were greviously flawed.

The test itself was not grievously flawed. The only obvious problem
was the one Scott mentioned--using different mikes for the two
conditions. But that only matters if you're comparing the results of
under the two conditions. Looked at individually, the two test
conditions tell us nothing.

The statistical analysis is another story.

I guess we can chalk this paper up as yet another "Proof by complex
statistical analysis" which defies common sense.

Agreed.

bob